!
`
`··,
`
`()
`
`',··-...
`<I)' 0
`<J) ~
`m
`(\J .0
`.0
`a:
`::l
`en
`N
`·en
`~
`()
`as w
`u :::l
`en
`~
`
`<J)
`<J)
`
`,j.
`1\::)
`('>
`
`,, .. PATENT NUMB~fL _
`:
`.
`•
`.,
`
`.·"l~·~;.:.-:'--;:
`
`8383345
`
`1111111 IIIII IIIII IIIII 1m llllllllllllllll\
`6363345
`
`u~s. UTILITY PATENT APPLICATION
`PATENT DATE
`O.I.P.E.
`' . 1'\AR 2 6 2\10!
`
`SCANNED
`
`Q.A.
`
`~­w
`
`SECTOR
`
`SUBCLASS
`
`-
`
`EXAMJ~t:R- .
`I
`,/f)··;,.. ··.:.• 'j. ') {
`...,_ ,_)\).(. v
`\....;
`
`·.
`
`'
`
`.. ~.::i:::J.: F~~E6WITH: D DISK (CRF) D FlyHE
`
`· /
`
`:.
`
`(Attached In pocket on right inside flap)
`
`j
`PREPARED AND;APPROVED FOR ISSUE
`1..--.--------.----'---------
`r.
`ISSUING .CLASSIFICATION
`
`• J
`
`.. I
`
`ORIGINAL
`
`CLAS$
`
`SUBCL~SS
`
`CLASS
`
`INTERNATIONAl CLASSIFICATION
`G-lo
`o'J...
`
`.~w .. ·
`. D TERMINAL
`.
`
`. DISCLAIMER
`
`D a) The term of this patent
`subsequent to _____ (date)
`has been disclaimed.
`D b) The term of this patent shall
`not extend beyond the expiration date
`of U.S Patent. No. ____ _
`
`D c) The terminal __ months of
`this patent have been disclaimed.
`
`D Continued on Issue Slip _Inside File J gr<at ~ -
`
`DRAWINGS
`
`Sheets Drwg.
`(0
`
`Figs. Drwg.
`(0
`
`Print Fig.
`
`..,...,
`CLAIMS"#\LLOWED
`......
`
`Print Claim for O.G.
`
`I
`
`_L ,. NOTICE OF ALLOWANCE MAILED
`~------~------~/
`(Assistant Examiner).
`jiifate)
`
`~ ;
`
`I D --(b·~ 6/
`
`Richemond Dorv"/
`P.timary Exam~:lr ;
`.~~ !D-7:0/
`·rh· \l·D 1
`
`(~gailnstruments Examine
`
`---:::--:--
`(Date)
`
`'
`
`(Primary Examiner)
`
`·
`
`(Data)
`
`ISSUE FEE
`
`Amount Due
`
`~ r;J( () ,· J'i)
`
`Date Paid
`
`,rit-
`
`ISSUE BATCH NUMBER
`
`WARNING:
`The information disclosed herein may be restricted. Unauthorized disclosure may be prohibited by the United States Code Title 35, Sections 122, 181' and 368.
`Possession outside the U.S. Patent & Trademark Office is restricted to authorized employees and contractors only.
`
`Form PT0·436A
`(Rev. 6/98)
`
`·<!!,;· oiL ••.
`(LABEL AREA)
`
`\
`
`Petitioner Apple Inc.
`Ex. 1002, p. 1
`
`

`
`I.
`
`tt·
`
`CONTENTS
`
`-····Date received
`(Incl. C. ofM.)
`or
`,
`Date Mailed ·
`
`. tMAR Vlr$"1AL9c..=li::.;.so: il_, _
`.
`
`' .
`
`Date received
`(Incl. C. of M.)
`_.or
`QaieMailed
`
`44. __ _____ _ _
`
`45 . - - - - - - - - - - -
`
`46 . - - - - - - - - , - - , - - - - - - -
`47. ________ _
`
`49.
`
`50.
`
`51.
`
`52.
`
`53.
`
`54.
`
`55.
`
`56.
`
`57.
`
`58.
`
`59.
`
`'
`
`60~
`
`61.
`
`62.
`
`63.
`
`64.
`
`65.
`
`66.
`
`67.
`
`68.
`
`69.
`ro.
`71.
`
`72.
`
`73.
`•' '. 74.
`' 75.
`
`76 .
`17.
`
`78.
`
`79.
`
`80.
`
`81.
`
`82.
`
`18. - - - ' - - - - - - - - - ' -
`19. _____ __ _ _
`
`20 . - , - - - - - - - , - - - - -
`
`22.
`
`23. - - - - - - - - ' - - -
`2~: - - - - - - - - -
`25 . - . , - - - - - - - - -
`26. ---"'·--· . .-_:;.-_·-··_' - - - - -
`27. __ ______ _
`
`28 . - - - - - - - , - - - -
`
`29 . - - - , - - - - - - - - -
`30. __ - . _____ _
`,,
`
`. 31. - - ' - - - - - ' - - - - - - - -
`
`' 32. ---,--,.,----~----'---­
`
`·33. - - - - , - - - -= - - - - - -
`
`34. - - ' - - - - - - - - - -
`
`35.- - - - - - - - - - - -
`36. ______ __:_ __
`
`..
`
`37 .. ____,.,----------...,-/ .. -
`v
`38. __ ______ _
`
`39. - - - - - - - - ' - - - -
`40. __ ______ _
`
`'('
`
`--~--
`
`- - - l . ' - -
`
`(LEFf OUTSIDE)
`
`Petitioner Apple Inc.
`Ex. 1002, p. 2
`
`

`
`SERIAL NUMBER
`
`FILING DATE
`
`CLASS
`
`GROUP ART UNIT
`
`ATTORNEY DOCKET NO.
`
`09/252,874
`
`02/18/99
`
`381
`
`2743
`
`670025-2800
`
`~----------------------~------------L-------------~--------------L------------------------j
`1-z
`JOSEPH MARASH, HAIFA, ISRAEL; BARUCH BERDUGO, KIRIAT-ATA 28000, ISRAEL.
`5
`
`:J
`0..
`0..
`<(
`
`l
`
`**CONTINUING DOMESTIC DATA*********************
`VERIFIED
`
`**371 (NAT'L STAGE) DATA*********************
`VERIFIED
`
`**FOREIGN APPLICATIONS************
`VERIFIED
`
`,/./
`. //
`//
`
`Dyes ~~o
`Sif'ATE OR
`Foreign Priority claimed
`35 USC 119 (a-d) conditions met Oye~_gLPo OMet lifter Allowance fOUNTRY
`62;\1_')
`/
`ILX
`Verified and Acknowledged
`tXBI ner·s nltiBIS
`
`nltlaiS
`
`SHEETS?'
`DRAWING
`10
`
`TOTAL
`CLAIMS
`49
`
`/
`
`INDE~DENT
`CLAIMS
`3
`
`(f)
`
`(f) w
`cc
`0
`0
`<(
`
`w
`....J
`1-
`i=
`"
`
`IF REQUIRED, FOREIGN FILING LICENSE GRANTED o;•~/99 ** SMA~NTITY ** /
`I
`
`THOMAS J KOWALSKI
`~~ROMMER LAWRENCE & HAUG
`745 FIFTH AVENUE
`NEW YORK NY 10151
`
`SYSTEM, METHOD AND APPARATUS FOR CANCELLING NOISE
`
`FILING FEE
`RECEIVED
`
`$664
`
`FEES: Authority has been given in Paper
`No.
`to charge/credit DEPOSIT ACCOUNT
`NO.
`for the following:
`
`0 All Fees
`0 1.16 Fees {Filing)
`
`8 1.17 Fees (Processing Ext. of time)
`1.18 Fees (Issue)
`0 Other------
`0 Credit
`
`Petitioner Apple Inc.
`Ex. 1002, p. 3
`
`

`
`FROMMER LAWRENCE & HAUG LLP
`745 Fifth Avenue
`New York, New York 10151
`Tel (212) 588-0800
`Fax (212) 588-0500
`PATENT APPLICATION TRANSMITTAL
`Date:
`Re:
`THE COMMISSIONER OF PATENTS AND TRADEMARKS
`Box PATENT APPLICATION
`Washington, D.C. 20231
`
`TO:
`
`Sir:
`
`February 18,
`670025-2800
`
`1999
`
`With reference to the filing in the United States Patent and
`Trademark Office of an application for patent in the name of:
`JOSEPH MARASH and BARUCH BERDUGO
`
`entitled: SYSTEM, METHOD AND APPARATUS FOR CANCELLING NOISE
`
`The
`K
`K
`K
`
`following are enclosed:
`Specification (22 pages) and One Page of Abstract (p. i)
`49 Claims (including ~ independent claims; pp. 23-31)
`10 Sheets of Drawings (Figs. 1, 2, 3, 4, 5, SA, 6, 7,
`8, 9l
`.
`--
`Unsigned Declaration and Power of Attorney (2 pages)
`The filing fee will be paid later, in response to a
`Notice to File Missing Parts. Kindly accord the
`application a February 18, 1998 f~ling date and address
`all communications to the undersigned at the address
`above.
`
`~~e~;ect~u~;;J_~ ~u~~~v.m~.t~';M?ed,.
`__
`t:D. ~
`./
`,}
`id.LP
`By :
`'CFt.·'«
`Thomas J. Kowalsk1, Reg. No. 32,147
`
`EXPRESS MAIL
`Mailing Label Number ~EM~2;=2~89~0~9.:,;13~1~U':?-S---:-=:-:c----
`Date of Deposit
`February 18, 1999
`I hereby certify that this paper or fee is being
`deposited with the United States Postal Service
`"Express Mail Post Office to Addressee" Service
`under 37 CFR 1.10 on the date indicated above and
`is add ssed to the Assistant Commissioner for Patents,
`Washi
`,,~D;C. f0231, BOX NEW PATENT APPLICATION
`
`VI ~~ 0n lL
`
`Petitioner Apple Inc.
`Ex. 1002, p. 4
`
`

`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`APPLICATION FOR LETTERS PATENT
`
`Title:
`
`SYSTEM, METHOD AND APPARATUS FOR CANCELLING NOISE
`
`Inventors:
`
`Joseph Marash, Baruch Berdugo
`
`~ pages specification and one page of Abstract (page i)
`
`_12_ Claim (3 Independent; on pages 23-31)
`
`_lQ_ sheets of Figs. (Figs. 1-5, SA, 6-9)
`
`EXPRESS MAIL
`Mailing Label Number ~09/..3'/ JIS.
`Date of Deposit c~"'' · ?g. /7'29
`I hereby certify that this !)ape or fee is bein~'
`deposited with the United States Postal Service
`"Express Mail Post Office to Addressee" Service
`under 37 CFR 1.10 on the date indicated above and
`is addressed to the Assistant Commissioner of
`Patents a
`radem rks, Washington, D.C. 20231
`l~ ~M.tA-Yl \
`
`(Signature of person mailing paper or fee)
`
`Thomas J. Kowalski
`Registration No. 32,147
`I. Marc Asperas
`Registration No. 37,274
`FROMMER LAWRENCE & HAUG LLP
`745 Fifth Avenue
`New York, New York 10151
`(212) 588-0800
`
`MARCA\2800.APP (IMA:car)
`
`Petitioner Apple Inc.
`Ex. 1002, p. 5
`
`

`
`PATENT
`670025-2800
`
`ABSTRACT OF THE DISCLOSURE
`
`A threshold detector precisely detects the positions of·
`
`the noise elements, even within continuous speech segments, by
`
`determining whether frequency spectrum elements, or bins, of the
`
`input signal are within a threshold set according to current and
`
`5
`
`future minimum values of the frequency spectrum elements.
`
`In
`
`addition, the threshold is continuously set and initiated within
`
`a predetermined period of time. The estimate magnitude of the
`
`input audio signal is obtained using a multiplying combination of
`
`the real and imaginary part of the input in accordance with the
`
`1
`
`higher and lower values between the real and imaginary part of
`
`the signal.
`
`In order to further,reduce instability of the
`
`spectral estimation, a two-dimensional smoothing is applied to
`
`"'
`
`the signal estimate using neighboring frequency bins and an
`
`exponential average over time. A filter multiplication effects
`
`l~n
`
`the subtraction thereby avoiding phase calculation difficulties
`
`and effecting full-wave rectification which further reduces
`
`artifacts. Since the noise elements are determined within
`
`continuous speech segments, the noise is canceled from the audio
`
`signal nearly continuously thereby providing excellent noise
`
`20
`
`cancellation characteristics. Residual noise reduction reduces
`
`the residual noise remaining after noise cancellation.
`
`Implementation may be effected in various noise canceling schemes
`
`including adaptive beamforming and noise cancellation using
`
`computer program applications installed as software or hardware.
`
`MARCA\2800.APP (IMA:car)
`
`i
`
`Petitioner Apple Inc.
`Ex. 1002, p. 6
`
`

`
`PATENT
`670025-2800
`
`RELATED APPLICATIONS INCORPORATED BY REFERENCE.
`
`The following applications and patent(s) are cited and
`
`hereby herein incorporated by reference: U.S. Patent Serial No.
`
`5
`
`09/130,923 filed August 6, 1998, U.S. Patent Serial No.
`
`09/055,709 filed April 7, 1998, U.S. Patent Serial No. 09/0~9,503
`
`filed April 13, 1998, U.S. Patent Serial No. 08/840,159 filed
`
`April 14, 1997, U.S. Patent Serial No. 09/130,923 filed August 6,
`
`1998, U.S. Patent Serial No. 08/672,899 now issued U.S. Patent
`
`lCi~':;
`
`No. 5,825,898 issued October 20, 1998. And, all documents cited
`
`herein are incorporated herein by reference, as are documents
`
`cited or referenced in documents cited herein.
`
`FIELD OF THE INVENTION.
`
`The present invention relates to noise cancellation and
`
`reduction and, more specifically, to noise cancellation and
`
`reduction using spectral subtraction.
`
`BACKGROUND OF THE INVENTION.
`
`Ambient noise added to speech degrades the performance
`
`20
`
`of speech processing algorithms. Such processing algorithms may
`
`include dictation, voice activation, voice compression and other
`
`systems.
`
`In such systems, it is desired to reduce the noise and
`
`improve the signal to noise ratio (S/N ratio) without effecting
`
`the speech and its characteristics.
`
`25
`
`Near field noise canceling microphones provide a
`
`MARCA\2800.APP (IMA:car)
`
`1
`
`Petitioner Apple Inc.
`Ex. 1002, p. 7
`
`

`
`PATENT
`670025-2800
`
`satisfactory solution but require that the microphone in the
`
`proximity of the voice source (e.g., mouth).
`
`In many cases, this
`
`is achieved by mounting the microphone on a boom of a headset
`
`which situates the microphone at the end of a boom proximate the
`
`5
`
`mouth of the wearer. However, the headset has proven to be
`
`either uncomfortable to wear or too restricting for operation in,
`
`for example, an automobile.
`
`Microphone array technology in general, and adaptive
`
`beamforming arrays in particular, handle severe directional
`
`1
`
`noises in the most efficient way. These systems map the noise
`
`field and create nulls towards the noise sources. The number of
`
`nulls is limited by the number of microphone elements and
`
`processing power. Such arrays have the benefit of hands-free
`
`operation without the necessity of a headset.
`
`However, when the noise sources are diffused, the
`
`performance of the adaptive system will be reduced to the
`
`performance of a regular delay and sum microphone array, which is
`
`not always satisfactory. This is the case where the environment
`
`is quite reverberant, such as when the noises are strongly
`
`20
`
`reflected from the walls of a room and reach the array from an
`
`infinite number of directions. Such
`
`is also the case in a car
`
`environment for some of the noises radiated from the car chassis.
`
`OBJECTS AND SUMMARY OF THE INVENTION
`
`25
`
`The spectral subtraction technique provides a solution
`
`MARCA\2800.APP (IMA:car)
`
`2
`
`Petitioner Apple Inc.
`Ex. 1002, p. 8
`
`

`
`PATENT
`670025-2800
`
`to further reduce the noise by estimating the noise magnitude
`
`spectrum of the polluted signal. The technique estimates the
`
`magnitude spectral level of the noise by measuring it during non-
`
`speech time intervals detected by a voice switch, and then
`
`5
`
`subtracting the noise magnitude spectrum from the signal. This
`
`method, described in detail in Suppression of Acoustic Noise in
`
`Speech Using Spectral Subtraction, (Steven F Boll, IEEE ASSP-27
`
`N0.2 April, 1979), achieves good results for stationary diffused
`
`noises that are not correlated with the speech signal. The
`
`1
`
`spectral subtraction method, however, creates artifacts,
`
`sometimes described as musical noise, that may reduce the
`
`performance of the speech algorithm (such as vocoders or voice
`
`activation) if the spectral subtraction is uncontrolled.
`
`In
`
`··
`
`addition, the spectral subtraction method assumes erroneously
`
`15iJ
`
`that the voice switch accurately detects the presence of speech
`
`and locates the non-speech time intervals. This assumption is
`
`reasonable-for off-line systems but difficult to achieve or
`
`obtain in real time systems.
`
`More particularly, the noise magnitude spectrum is
`
`20
`
`estimated by performing an FFT of 256 points of the non-speech
`
`time intervals and computing the energy of each frequency bin.
`
`The FFT is performed after the time domain signal is multiplied
`
`by a shading window (Hanning or other) w~th an overlap of 50%.
`
`The energy of each frequency bin is averaged with neighboring FFT
`
`25
`
`time frames. The number of frames is not determined but depends
`
`MARCA\2800.APP (IMA:car)
`
`4 3
`
`Petitioner Apple Inc.
`Ex. 1002, p. 9
`
`

`
`PATENT
`670025-2800
`
`on the stability of the noise. For a stationary noise, it is
`
`preferred that many frames are averaged to obtain better noise
`
`estimation. For a non-stationary noise, a long averaging may be
`
`harmful. Problematically, there is no means to know a-priori
`
`5
`
`whether the noise is stationary or non-stationary.
`
`Assuming the noise magnitude spectrum estimation is
`
`calculated, the input signal is multiplied by a shading window
`
`(Hanning or other), an FFT is performed (256 points or other)
`
`with an overlap of 50% and the magnitude of each bin is averaged
`
`1
`
`over 2-3 FFT frames. The noise magnitude spectrum is then
`
`subtracted from the signal magnitude.
`
`If the result is negative,
`
`the value is replaced by a zero {Half Wave Rectification) .
`
`It is
`
`recommended, however,
`
`to further reduce the residual noise
`
`"
`
`present during non-speech intervals by replacing low values with
`
`1$1!
`
`a minimum value (or zero) or by attenuating the residual noise by
`
`30dB. The resulting output is the noise free magnitude spectrum.
`
`The spectral complex data is reconstructed by applying
`
`the phase information of the relevant bin of the signal's FFT
`
`with the noise free magnitude. An IFFT process is then performed
`
`20
`
`on the complex data to obtain the noise free time domain data.
`
`The time domain results are overlapped and summed with the
`
`previous frame's results to compensate for the overlap process of
`
`the FFT.
`
`There are several problems associated with the system
`
`25
`
`described. First, the system assumes that there is a prior
`
`MARCA\2800.APP (IMA:car)
`
`Petitioner Apple Inc.
`Ex. 1002, p. 10
`
`

`
`PATENT
`670025-2800
`
`knowledge of the speech and non-speech time intervals. A voice
`
`switch is not practical
`
`to detect those periods. Theoretically,
`
`a voice switch detects the presence of the speech by measuring
`
`the energy level and comparing it to a threshold.
`
`If the
`
`5
`
`threshold is too high, there is a risk that some voice time
`
`intervals might be regarded as a non-speech time interval and the
`
`system will regard voice information as noise. The result is
`
`voice distortion, especially in poor signal to noise ratio cases.
`
`If, on the other hand, the threshold is too low, there is a risk
`
`1
`
`that the non-speech intervals will be too short especially in
`
`poor signal to noise ratio cases and in cases where the voice is
`
`continuous with little intermission.
`
`Another problem is that the magnitude calculation of
`
`the FFT result is quite complex. This involves square and square
`
`l~U
`
`root calculations which are very expensive in terms of
`
`computation load. Yet another problem is the ~ssociation of the
`
`phase information to the noise free magnitude spectrum in order
`
`to~obtain the information for the IFFT. This process requires
`
`the calculation of the phase, the storage of the information, and
`
`20
`
`applying the information to the magnitude data - all are
`
`expensive in terms of. computation and memory requirements.
`
`Another problem is the estimation of the noise spectral
`
`magnitude. The FFT process is a poor and unstable estimator of
`
`energy. The averaging-over-time of frames contributes
`
`25
`
`insufficiently to the stability. Shortening the length of the
`
`MARCA\2800.APP (IMA:car)
`
`5
`
`Petitioner Apple Inc.
`Ex. 1002, p. 11
`
`

`
`PATENT
`670025-2800
`
`FFT results in a wider bandwidth of each bin and better stabili.ty
`
`but reduces the performance of the system. Averaging-over-time,
`
`moreover, smears the data and, for this reason, cannot be
`
`extended to more than a few frames. This means that the noise
`
`5
`
`estimation process proposed is not sufficiently stable.
`
`It is therefore an object of this invention to provide
`
`a spectral subtraction system that has a simple, yet efficient
`
`mechanism, to estimate the noise magnitude spectrum even in poor
`
`signal-to-noise ratio situations and in continuous fast speech
`
`1
`
`cases.
`
`It is another object of this invention to provide an
`
`efficient mechanism that can perform the magnitude estimation
`
`with little cost, and will overcome the problem of phase
`
`association.
`
`1~
`
`It is yet another object of this invention to provide a
`
`stable mechanism to estimate the noise spectral magnitude without
`
`the smearing of the data.
`
`In accordance with the foregoing objectives, the
`
`present invention provides a system that correctly determines the
`
`20
`
`non-speech segments of the audio signal thereby preventing
`
`erroneous processing of the noise canceling signal during the
`
`speech segments.
`
`In the preferred embodiment, the present
`
`invention obviates the need for a voice switch by precisely
`
`determining the non-speech segments using a separate threshold
`
`25
`
`detector for each frequency bin. The threshold detector
`
`MARCA\2800.APP (IMA:car)
`
`6 A
`. . I
`!
`
`Petitioner Apple Inc.
`Ex. 1002, p. 12
`
`

`
`PATENT
`670025-2800
`
`precisely detects the positions of the noise elements, even
`
`within continuous speech segments, by determining whether
`
`frequency spectrum elements, or bins, of the input signal are
`
`within a threshold set according to a minimum value of the
`
`5
`
`frequency spectrum elements over a preset period of time. More
`
`precisely, current and future minimum values of the frequency
`
`spectrum elements. Thus, for each syllable, the energy of the
`
`noise elements is determined by a separate threshold
`
`determination without examination of the overall signal energy
`
`1
`
`thereby providing good and stable estimation of the noise.
`
`In
`
`addition, the system preferably sets the threshold continuously
`
`and resets the threshold within q predetermined period of time
`
`for example, five seconds.
`
`In order to reduce complex calculations, it is
`
`lSiJ
`
`preferred in the present invention to obtain an estimate of the
`
`magnitude of the input audio signal using a multiplying
`
`combination of the real and imaginary parts of the input in
`
`accordance with, for example, the higher and the lower values of
`
`the real and imaginary parts of the signal.
`
`In order to further
`
`20
`
`reduce instability of the spectral estimation, a two-dimensional
`
`(2D) smoothing proces9 is applied to the signal estimation. A
`
`two-step smoothing function using first neighboring frequency
`
`bins in each time frame then applying an exponential time average
`
`effecting an average over time for each frequency bin produces
`
`25
`
`excellent results.
`
`MARCA\2800.APP (IMA:car)
`
`1 7
`
`Petitioner Apple Inc.
`Ex. 1002, p. 13
`
`

`
`PATENT
`670025-2800
`
`In order to reduce the complexity of determining the
`
`phase of the frequency bins during subtraction to thereby align
`
`the phases of the subtracting elements, the present invention
`
`applies a filter multiplication to effect the subtraction. The
`
`5
`
`filter function, a Weiner filter function for example, or an
`
`approximation of the Weiner filter is multiplied by the complex
`
`data of the frequency domain audio signal. The filter function
`
`may effect a full-wave rectification, or a half-wave
`
`rectification for otherwise negative results of the subtraction
`
`1
`
`process or simple subtraction.
`
`It will be appreciated that,
`
`since the noise elements are determined within continuous speech
`
`segments, the noise estimation is accurate and it may be canceled
`
`from the audio signal continuously providing excellent noise
`
`cancellation characteristics.
`
`1~
`
`The present invention also provides a residual noise
`
`reduction process for reducing the residual noise remaining after
`
`noise cancellation. The residual noise is reduced by zeroing the
`
`non-speech segments, e.g., within the continuous speech, or
`
`decaying the non-speech segments. A voice switch may be used or
`
`20
`
`another threshold detector which detects the non-speech segments
`
`in the time-domain.
`
`The present invention is applicable with various noise
`
`canceling systems including, but not limited to, those systems
`
`described in the U.S. patent applications incorporated herein by
`
`25
`
`reference. The present invention, for example, is applicable
`
`MARCA\2800.APP (IMA:car)
`
`8
`
`Petitioner Apple Inc.
`Ex. 1002, p. 14
`
`

`
`with the adaptive beamforming array.
`
`In addition, the present
`
`invention may be embodied as a computer program for driving a
`
`computer processor either installed as application software or as
`
`PATENT
`670025-2800
`
`hardware.
`
`5
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Other objects, features and advantages according to the
`
`present invention will become apparent from the following
`
`detailed description of the illustrated embodiments when read in
`
`1
`
`conjunction with the accompanying drawings in which corresponding
`
`components are identified by the same reference numerals.
`
`Fig. 1 illustrates the 'present inventioni
`
`Fig. 2 illustrates the noise processing of the present
`
`inventioni
`
`1~
`
`Fig. 3 illustrates the noise estimation processing of
`
`the present inventioni
`
`Fig. 4 illustrates the subtraction processing of the
`
`present inventioni
`
`Fig. 5 illustrates the residual noise processing of the
`
`20
`
`present inventioni
`
`Fig. SA illustrates a variant of the residual noise
`
`processing of the present inventioni
`
`Fig. 6 illustrates a flow diagram of the present
`
`inventioni
`
`25
`
`Fig. 7 illustrates a flow diagram of the present
`
`MARCA\2800.APP (IMA:car)
`
`9
`
`J
`
`Petitioner Apple Inc.
`Ex. 1002, p. 15
`
`

`
`PATENT
`670025-2800
`
`invention;
`
`Fig. 8 illustrates a flow diagram of the present
`
`invention; and
`
`Fig. 9 illustrates a flow diagram of the present
`
`5
`
`invention.
`
`DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
`
`Figure 1 illustrates an embodiment of the present
`
`invention 100. The system receives a digital audio signal at
`
`1
`
`input 102 sampled at a frequency which is at least twice the
`
`bandwidth of the audio signal.
`
`In one embodiment, the signal is
`
`derived from a microphone signal that has 'been processed through
`'
`an analog front end, A/D converter and a decimation filter to
`
`obtain the required sampling frequency.
`
`In another embodiment,
`
`15U
`
`the input is taken from the output of a beamformer or even an
`
`adaptive beamformer.
`
`In that case the signal has been processed
`
`to eliminate noises arriving from directions other than the
`
`desired one leaving mainly noises originated from the same
`
`direction of the desired one.
`
`In yet another embodiment, the
`
`20
`
`input signal can be obtained from a sound board when the
`
`processing is implemented on a PC processor or similar computer
`
`processor.
`
`The input samples are $tored in a temporary buffer 104
`
`of 256 points. When the buffer is full, the new 256 points are
`
`25
`
`combined in a combiner 106 with the previous 256 points to
`
`MARCA\2800.APP (IMA:car)
`
`10
`
`I I
`
`Petitioner Apple Inc.
`Ex. 1002, p. 16
`
`

`
`PATENT
`670025-2800
`
`provide 512 input points. The 512 input points are multiplied by
`
`multiplier 108 with a shading window with the length of 512 .
`
`points. The shading window contains coefficients that are
`
`multiplied with the input data accordingly. The shading window
`
`5
`
`can be Hanning or other and it serves two goals: the first is to
`
`smooth the transients between two processed blocks (together with
`
`the overlap process) ; the second is to reduce the side lobes in
`
`the frequency domain and hence prevent the masking of low energy
`
`tonals by high energy side lobes. The shaded results are
`
`1
`
`converted to the frequency domain through an FFT (Fast Fourier
`
`Transform) processor 110. Other lengths of the FFT samples (and
`
`accordingly input buffers) are pqssible including 256 points or
`
`1024 points.
`
`The FFT output is a complex yector of 256 significant
`
`15'U
`
`points (the other 256 points are an anti-symmetric replica of the
`
`first 256 points) . The points are processed in the noise
`
`processing block 112(200) which includes the noise magnitude
`
`esEimation for each frequency bin -
`
`the subtraction process that
`
`estimates the noise-free complex value for each frequency bin and
`
`20
`
`the residual noise reduction process. An IFFT (Inverse Fast
`
`Fourier Transform) processor 114 performs the Inverse Fourier
`
`Transform on the complex noise free data to provide 512 time
`
`domain points. The first 256 time domain points are summed by
`
`the summer 116 with the previous last 256 data points to
`
`25
`
`compensate for the input overlap and shading process and output
`
`MARCA\2800.APP (IMA:car)
`
`11
`
`Petitioner Apple Inc.
`Ex. 1002, p. 17
`
`

`
`PATENT
`670025-2800
`
`at output terminal 118. The remaining 256 points are saved for
`
`the next iteration.
`
`It will be appreciated that, while specific transforms
`
`are utilized in the preferred embodiments, it is of course
`
`5
`
`understood that other transforms may be applied to the present
`
`invention to obtain the spectral noise signal.
`
`Figure 2 is a detailed description of the noise
`
`processing block 200(112). First, each frequency bin (n) 202
`
`magnitude is estimated. The straight forward approach is to
`
`1
`
`estimate the magnitude by calculating:
`
`Y (n) = ((Real (n) F+ (Imag (n) F) -2
`
`In order to save processing time and complexity the
`
`15'U
`
`signal magnitude (Y) is estimated by an estimator 204 using an
`
`approximation formula instead:
`
`Y(n)
`
`Max [I Real (n), Imag (n) I 1 +0. 4 * Min [I Real (n), Imag (n) I 1
`
`20
`
`In order to reduce the instability of the spectral
`
`estimation, which typically plagues the FFT Process (ref[21
`
`Digital Signal Processing, Oppenheim Schafer, Prentice Hall P.
`
`542545), the present invention implements a 2D smoothing process.
`
`Each bin is replaced with the average of its value and the two
`
`25
`
`neighboring bins' value (of the same time frame) by a first
`
`MARCA\2800.APP (IMA:car)
`
`12
`
`/3
`
`Petitioner Apple Inc.
`Ex. 1002, p. 18
`
`

`
`PATENT
`670025-2800
`
`averager 206.
`
`In addition, the smoothed value of each smoothed
`
`bin is further smoothed by a second averager 208 using a time
`
`exponential average with a time constant of 0.7 (which is the
`
`equivalent of averaging over 3 time frames) . The 2D-smoothed
`
`5
`
`value is then used by two processes -
`
`the noise estimation
`
`process by noise estimation processor 212(300) and the
`
`subtraction process by subtractor 210. The noise estimation
`
`process estimates the noise at each frequency bin and the result
`
`is used by the noise subtraction process. The output of the
`
`1
`
`noise subtraction is fed into a residual noise reduction
`
`processor 216 to further reduce the noise.
`
`In one embodiment,
`
`the time domain signal is also u~ed by the residual noise process
`
`216 to determine the speech free segments. The noise free signal
`
`is moved to the IFFT process to obtain the time domain output
`
`EL~
`
`218.
`
`Figure 3 is a detailed description of the noise
`
`estimation processor 300(212). Theoretically, the noise should
`
`be~estimated by taking a long time average of the signal
`
`magnitude (Y) of non-speech time intervals. This requires that a
`
`20
`
`voice switch be used to detect the speech/non-speech intervals.
`
`However, a too-sensitive a switch may result in the use of a
`
`speech signal for the noise estimation which will defect the
`
`voice signal. A less sensitive switch, on the other hand, may
`
`dramatically reduce the length of the noise time intervals
`
`25
`
`(especially in continuous speech cases) and defect the validity
`
`MARCA\2800.APP (IMA:car)
`
`13
`
`Petitioner Apple Inc.
`Ex. 1002, p. 19
`
`

`
`PATENT
`670025-2800
`
`of the noise estimation.
`
`In the present invention, a separate adaptive threshold
`
`is implemented for each frequency bin 302. This allows the
`
`location of noise elements for each bin separately without the
`
`5
`
`examination of the overall signal energy. The logic behind this
`
`method is that, for each syllable, the energy may appear at
`
`different frequency bands. At the same time, other frequency
`
`bands may contain noise elements.
`
`It is therefore possible to
`
`apply a non-sensitive threshold for the noise and yet locate many
`
`non-speech data points for each bin, even within a continuous
`
`speech case. The advantage of this method is that it allows the
`
`collection of many noise segments for a good and stable
`
`estimation of the noise, even within continuous speech segments.
`
`In the threshold determination process, for each
`
`lgU
`
`frequency bin, two minimum values are calculated. A future
`
`minimum value is initiated every 5.seconds at 304 with the value
`
`of the current magnitude (Y(n)) and replaced with a smaller
`
`minimal value over the next 5 seconds through the following
`
`process. The future minimum value of each bin 1s compared with
`
`20
`
`the current magnitude value of the signal.
`
`If the current
`
`magnitude is smaller than the future minimum, the future minimum
`
`is replaced with the magnitude which becomes the new future
`
`minimum.
`
`At the same time, a current minimum value is calculated
`
`25
`
`at 306. The current minimum is initiated every 5 seconds with
`
`MARCA\2800.APP (IMA:car)
`
`14
`
`Petitioner Apple Inc.
`Ex. 1002, p. 20
`
`

`
`PATENT
`670025-2800
`
`the value of the future minimum that was determined over the
`
`previous 5 seconds and follows the minimum value of the signal
`
`for the next 5 seconds by comparing its value with the current
`
`magnitude value. The current minimum value is used by the
`
`5
`
`subtraction process, while the future minimum is used for the
`
`initiation and refreshing of the current minimum.
`
`The noise estimation mechanism of the present invention
`
`ensures a tight and quick estimation of the noise value, with
`
`limited memory of the process (5 seconds), while preventing a too
`
`1
`
`high an estimation of the noise.
`
`Each bin's magnitude (Y(n)) is compared with four times
`
`the current minimum value of that bin by comparator 308 - which
`
`serves as the adaptive threshold for that bin.
`
`If the magnitude
`
`"
`
`is within the range (hence below the threshold) , it is allowed as
`
`151"'~
`
`noise and used by an exponential averaging unit 310 that
`
`determines the level of the noise 312 of that frequency.
`
`If the
`
`magnitude is above the threshold it is rejected for the noise
`
`estimation. The time constant for the exponential averaging is
`
`typically 0.95 which may be interpreted as taking the average of
`
`20
`
`the last 20 frames. The threshold of 4*minimum value may be
`
`changed for some applications.
`
`Figure 4 is a detailed d~scription of the subtraction
`
`processor 400(210).
`
`In a straight forward approach, the value of
`
`the estimated bin noise magnitude is subtracted from the current
`
`25
`
`bin magnitude. The phase of the current bin is calculated and
`
`MARCA\2800.APP (IMA:car)
`
`15
`
`Petitioner Apple Inc.
`Ex. 1002, p. 21
`
`

`
`PATENT
`670025-2800
`
`used in conjunction with the result of the subtraction to obtain
`
`the Real and Imaginary parts of the result. This approach is
`
`very expensive in terms of processing and memory because it
`
`requires the calculation of the Sine and Cosine arguments of the
`
`5
`
`complex vector with consideration of the 4 quarters where the
`
`complex vector may be positioned. An alternative approach used
`
`in this present invention is to use a Filter approach. The
`
`subtraction is interpreted as a filter multiplication performed
`
`by filter 402 where H (the filter co