(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2008/0240458 A1
`(43) Pub. Date:
`Oct. 2, 2008
`Goldstein et al.
`
`US 20080240458A1
`
`(54) METHOD AND DEVICE CONFIGURED FOR
`SOUND SIGNATURE DETECTION
`
`(75) Inventors:
`
`Steven W. Goldstein, Delray
`Beach, FL (US); Mark A.
`Clements, Lilburn, GA (US); Marc
`A. Boillot, Plantation, FL (US)
`
`Correspondence Address:
`GREENBERG TRAURIG, LLP
`2101 L Street, N.W., Suite 1000
`Washington, DC 20037 (US)
`
`(73) Assignee:
`
`PERSONICS HOLDINGS INC.,
`Boca Raton, FL (US)
`
`(21) Appl. No.:
`
`11/966,457
`
`(22) Filed:
`
`Dec. 28, 2007
`
`
`
`Related U.S. Application Data
`(60) Provisional application No. 60/883,013, filed on Dec.
`31, 2006.
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`A6IF II/06
`(52) U.S. Cl. .......................................................... 381/72
`(57)
`ABSTRACT
`At least one exemplary embodiment is directed to a method
`for personalized listening which can be used with an earpiece
`is provided that can include capturing ambient Sound from an
`Ambient Sound Microphone (ASM) of an earpiece partially
`or fully occluded in an ear canal, monitoring the ambient
`Sound for a target Sound, and adjusting by way of an Ear Canal
`Receiver (ECR) in the earpiece a delivery of audio to an ear
`canal based on a detected target Sound. A Volume of audio
`content can be adjusted upon the detection of a target Sound,
`and an audible notification can be presented to provide a
`warning.
`
`Exhibit 1022
`Page 01 of 14
`
`

`

`Patent Application Publication
`
`Oct. 2, 2008 Sheet 1 of 5
`
`US 2008/0240458A1
`
`130
`
`Ambient : Sound
`Sound ; Signature :
`Detection :
`- - - - - - - - - - - - -
`
`110
`
`
`
`
`
`Interface
`212
`
`
`
`Audio Content
`(e.g., music, Cell
`phone, voice
`mail)
`
`100
`
`FIG. 2
`
`Exhibit 1022
`Page 02 of 14
`
`

`

`Patent Application Publication
`
`Oct. 2, 2008 Sheet 2 of 5
`
`US 2008/0240458A1
`
`Monitoring and Target Detection
`
`Monitoring the environment for
`target Sounds
`302
`
`
`
`
`
`
`
`
`
`Generating an audible alarm
`within the ear canal identifying
`the detected sound signature
`(e.g., Sound byte, text-to
`Speech, "ambulance
`approaching")
`
`308
`
`Detecting the target sounds
`Within the environment
`based on sound signatures
`304
`
`Adjusting audio delivered to
`to the ear canal in view of a
`detected target sound
`306
`
`
`
`
`
`
`
`
`
`
`
`Sending a message to a mobile
`device identifying the detected
`Sound signature (e.g., "alarm
`Sounding")
`310
`
`
`
`300
`FIG. 3
`
`
`
`Signature
`Sound Pass
`Through
`Mode
`(SSPTM)
`
`
`
`
`
`
`
`
`
`
`
`
`
`Mode Selection
`
`
`
`
`
`
`
`Signature
`Sound
`Replacement
`Mode (SSRM)
`
`Signature
`Sound Boost
`Mode
`(SSBM)
`
`Signature
`Sound
`Attenuation
`Mode
`(SSAM)
`
`
`
`
`
`
`
`
`
`
`
`
`
`Signature
`Sound
`Rejection
`Mode
`(SSRM)
`
`
`
`Exhibit 1022
`Page 03 of 14
`
`

`

`Patent Application Publication
`
`Oct. 2, 2008 Sheet 3 of 5
`
`US 2008/0240458A1
`
`
`
`
`
`520
`
`at 504
`
`508
`
`User Defined
`database
`
`Generate Speec
`h
`Recognition Mod
`es
`506
`
`510
`
`On-line
`database
`
`Generate Sound
`Signature Models
`
`
`
`Input signal from Ambient
`Sound Microphone (ASM)
`522
`Data Buffer
`524
`Signal Conditioning (e.g.,
`Noise Reduction, Noise gate)
`526
`Threshold Decision
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`530
`
`540
`
`Pattern Recognition to detect
`Sound signatures in input
`signal using Models
`
`Speech Recognition to
`recognize speech in input
`signal using Models
`
`Sound Source
`range/bearing
`analysis
`
`Sound Source
`speed
`analysis
`
`Key word
`detection
`(e.g. "help")
`
`542
`
`532
`Target signal
`detector (e.g.,
`horn, alarm)
`
`552
`
`Prioritizing
`
`554
`
`
`
`
`
`Operating
`
`550
`
`Selection
`
`an
`
`F G 5
`
`Exhibit 1022
`Page 04 of 14
`
`

`

`Patent Application Publication
`
`Oct. 2, 2008 Sheet 4 of 5
`
`US 2008/0240458A1
`
`Managing Audio Delivery
`
`Audio Interface receives audio
`content from a media player 602
`
`Audio content delivered to the
`user's ear Canal
`604
`
`
`
`600
`FIG. 6
`
`Signature
`Detected?
`608
`
`
`
`NO
`
`Processor monitors ambient
`Sound in the environment
`captured at the ASM
`
`60
`
`Processor determines a priority
`of the detected signature 610
`varara
`
`Processor selectively manages at
`least a portion of the audio Content
`based on the priority
`612
`
`
`
`Processor sends a message to a
`device operated by the user to
`visually display the notification 616
`
`Processor presents an audible
`notification to the user
`614
`
`
`
`
`
`
`
`Buffer data
`711
`
`Extract features
`712
`
`Compare models
`713
`
`Classify target Sound
`715
`
`
`
`
`
`Distortion Metric
`714.
`
`
`
`
`
`
`
`
`
`Generate alarm
`716
`
`FIG. 7
`
`Target Sounds
`database 718
`
`Exhibit 1022
`Page 05 of 14
`
`

`

`Patent Application Publication
`
`Oct. 2, 2008 Sheet 5 of 5
`
`US 2008/0240458A1
`
`Processor
`
`
`
`PHL
`Filtering
`
`Audio
`Content
`
`800
`FIG. 8
`
`Exhibit 1022
`Page 06 of 14
`
`

`

`US 2008/0240458 A1
`
`Oct. 2, 2008
`
`METHOD AND DEVICE CONFIGURED FOR
`SOUND SIGNATURE DETECTION
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`0001. This Application is a Non-Provisional and claims
`the priority benefit of Provisional Application No. 60/883,
`013 filed on Dec. 31, 2006, the entire disclosure of which is
`incorporated herein by reference.
`
`FIELD
`0002 The present invention relates to a device that moni
`tors target (e.g. warning) sounds, and more particularly,
`though not exclusively, to an earpiece and method of operat
`ing an earpiece that detects target Sounds.
`
`BACKGROUND
`0003 Excess noise exposure can generate auditory
`fatigue, possibly comprising a person's listening abilities. On
`a daily basis, people are exposed to various environmental
`Sounds and noises within their environment, Such as the
`Sounds from traffic, construction, and industry. Some of the
`Sounds in the environment may correspond to warnings, such
`as those associated with an alarm or siren. A person that can
`hear the warning sounds can generally react in time to avoid
`danger. In contrast, a person that cannot adequately hear the
`warning Sounds, or whose hearing faculties have been com
`promised due to auditory fatigue, may be susceptible to dan
`ger.
`0004 Environmental noise can mask warning sounds and
`impair a person's judgment. Moreover, when people wear
`headphones to listen to music, or engage in a call using a
`telephone, they can effectively impair their auditory judg
`ment and their ability to discriminate between sounds. With
`Such devices, the person is immersed in the audio experience
`and generally less likely to hear target Sounds within their
`environment. In some cases, the user may even turn up the
`Volume to hear their personal audio over environmental
`noises. This can put the user in a compromising situation
`since they may not be aware of target Sounds in their environ
`ment. It also puts them at high sound exposure risk, which can
`potentially cause long term hearing damage.
`0005. A need therefore exists for enhancing the user's
`ability to hear target sounds in their environment without
`compromising his hearing.
`
`SUMMARY
`0006. At least one exemplary embodiment is directed to a
`method and device for Sound signature detection.
`0007. In at least one exemplary embodiment, an earpiece,
`can include an Ambient Sound Microphone (ASM) config
`ured to capture ambient sound, at least one Ear Canal
`Receiver (ECR) configured to deliver audio to an ear canal,
`and a processor operatively coupled to the ASM and the at
`least one ECR to monitor target Sounds in the ambient sound.
`Target (e.g., warning) sounds can be amplified, attenuated, or
`reproduced and reported to the userby way of the ECR. As an
`example, the target (e.g., warning) sound can be an alarm, a
`horn, a voice, or a noise. The processor can detect Sound
`signatures in the ambient sound to identify the target (e.g.,
`warning) sounds and adjust the audio delivered to the ear
`canal based on detected Sound signatures.
`
`0008. In a second exemplary embodiment, a method for
`personalized listening Suitable for use with an earpiece is
`provided. The method can include capturing ambient Sound
`from an Ambient Sound Microphone (ASM) of an earpiece
`that is partially or fully occluded in an ear canal, monitoring
`the ambient sound for a target Sound, and adjusting by way of
`an Ear Canal Receiver (ECR) in the earpiece a delivery of
`audio to an ear canal based on a detected target Sound. The
`method can include passing, amplifying, attenuating, or
`reproducing the target Sound for delivery to the ear canal.
`0009. In a third exemplary embodiment a method for per
`Sonalized listening Suitable for use with an earpiece can
`include the steps of capturing ambient Sound from an Ambi
`ent Sound Microphone (ASM) of an earpiece that is partially
`or fully occluded in an ear canal, detecting a sound signature
`within the ambient sound that is associated with a target
`Sound, and mixing the target Sound with audio content deliv
`ered to the earpiece in accordance with a priority of the target
`Sound. A direction and speed of a sound source generating the
`target Sound can be determined, and presented as a notifica
`tion to a user of the earpiece. The method can include detect
`ing a spoken utterance in the ambient sound that corresponds
`to a verbal warning or help request.
`0010. In a fourth exemplary embodiment a method for
`Sound signature detection can include capturing ambient
`sound from an Ambient Sound Microphone (ASM) of an
`earpiece, and receiving a directive to learn a Sound signature
`within the ambient sound. The method can include receiving
`a voice command or detecting a user interaction with the
`earpiece to initiate the step of capturing and learning. A Sound
`signature can be generated for a target Sound in the environ
`ment and saved to a memory locally on the earpiece or
`remotely on a server.
`0011. In a fifth exemplary embodiment a method for per
`Sonalized listening can include capturing ambient sound from
`an Ambient Sound Microphone (ASM) of an earpiece that is
`partially or fully occluded in an ear canal, detecting a Sound
`signature within the ambient Sound that is associated with a
`target Sound, and mixing the target Sound with audio content
`delivered to the earpiece in accordance with a priority of the
`target sound and a personalized hearing level (PHL). The
`method can include retrieving from a database learned mod
`els, comparing the sound signature to the learned models, and
`identifying the target Sound from the learned models in view
`of the comparison. Auditory queues in the target Sound can be
`enhanced relative to the audio content based on a spectrum of
`the ambient sound captured at the ASM. A perceived direction
`of a sound source generating the target Sounds can be spatial
`ized using Head Related Transfer Functions (HRTFs).
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0012 FIG. 1 is a pictorial diagram of an earpiece in accor
`dance with an exemplary embodiment;
`0013 FIG. 2 is a block diagram of the earpiece in accor
`dance with an exemplary embodiment;
`0014 FIG. 3 is a flowchart of a method for ambient sound
`monitoring and target detection in accordance with an exem
`plary embodiment;
`0015 FIG. 4 illustrates earpiece modes in accordance with
`an exemplary embodiment;
`0016 FIG. 5 illustrates a flowchart of a method for sound
`signature detection in accordance with an exemplary embodi
`ment;
`
`Exhibit 1022
`Page 07 of 14
`
`

`

`US 2008/0240458 A1
`
`Oct. 2, 2008
`
`0017 FIG. 6 is a flowchart of a method for managing audio
`delivery based on detected Sound signatures in accordance
`with an exemplary embodiment;
`0018 FIG. 7 is a flowchart for sound signature detection in
`accordance with an exemplary embodiment; and
`0019 FIG. 8 is a pictorial diagram for mixing ambient
`Sounds and target Sounds with audio content in accordance
`with an exemplary embodiment.
`
`DETAILED DESCRIPTION
`0020. The following description of at least one exemplary
`embodiment is merely illustrative in nature and is in no way
`intended to limit the invention, its application, or uses.
`0021
`Processes, techniques, apparatus, and materials as
`known by one of ordinary skill in the relevant art may not be
`discussed in detail but are intended to be part of the enabling
`description where appropriate, for example the fabrication
`and use of transducers. Additionally in at least one exemplary
`embodiment the sampling rate of the transducers can be var
`ied to pick up pulses of Sound, for example less than 50
`milliseconds.
`0022. In all of the examples illustrated and discussed
`herein, any specific values, for example the Sound pressure
`level change, should be interpreted to be illustrative only and
`non-limiting. Thus, other examples of the exemplary embodi
`ments could have different values.
`0023 Note that similar reference numerals and letters
`refer to similar items in the following figures, and thus once
`an item is defined in one figure, it may not be discussed for
`following figures.
`0024 Note that herein when referring to correcting or
`preventing an error or damage (e.g., hearing damage), a
`reduction of the damage or error and/or a correction of the
`damage or error are intended.
`0025. At least one exemplary embodiment of the invention
`is directed to an earpiece for ambient Sound monitoring and
`target detection. Reference is made to FIG. 1 in which an
`earpiece device, generally indicated as earpiece 100, is con
`structed in accordance with at least one exemplary embodi
`ment of the invention. Earpiece 100 includes an Ambient
`Sound Microphone (ASM) 110 to capture ambient sound, an
`Ear Canal Receiver (ECR) 120 to deliver audio to an earcanal
`140, and an ear canal microphone (ECM) 130 to assess a
`sound exposure level within the ear canal. The earpiece 100
`can partially or fully occlude the ear canal 140 to provide
`various degrees of acoustic isolation.
`0026. The earpiece 100 can actively monitor a sound pres
`Sure level both inside and outside an ear canal and enhance
`spatial and timbral sound quality to ensure safe reproduction
`levels. The earpiece 100 in various exemplary embodiments
`can provide listening tests, filter sounds in the environment,
`monitor target Sounds in the environment, present notifica
`tions based on identified target Sounds, adjust audio content
`levels with respect to ambient sound levels, and filter sound in
`accordance with a Personalized Hearing Level (PHL). The
`earpiece 100 is suitable for use with users having healthy or
`abnormal auditory functioning. The earpiece 100 can be an in
`the ear earpiece, behind the ear earpiece, receiver in the ear,
`open-fit device, or any other Suitable earpiece type. Accord
`ingly, the earpiece 100 can be partially or fully occluded in the
`ear canal.
`0027. As part of its operation, the earpiece 100 can gener
`ate an Ear Canal Transfer Function (ECTF) to model the ear
`canal 140 using ECR 120 and ECM 130. The ECTF can be
`
`used to establish a personalized hearing level profile. The
`earpiece 100 can also determine a sealing profile with the
`user's ear to compensate for any sound leakage. In one con
`figuration, the earpiece 100 can provide personalized full
`band width general audio reproduction within the user's ear
`canal via timbral equalization based on the ECTF to account
`for a user's hearing sensitivity. The earpiece 100 also provides
`Sound Pressure Level dosimetry to estimate sound exposure
`of the ear and associated recovery times from excessive Sound
`exposure. This permits the earpiece 100 to safely administer
`and monitor Sound exposure to the ear.
`0028 Referring to FIG. 2, a block diagram of the earpiece
`100 in accordance with an exemplary embodiment is shown.
`As illustrated, the earpiece 100 can include a processor 206
`operatively coupled to the ASM 110, ECR 120, and ECM 130
`via one or more Analog to Digital Converters (ADC) 202 and
`Digital to Analog Converters (DAC) 203. The processor 206
`can monitor the ambient sound captured by the ASM 110 for
`target Sounds in the environment, such as an alarm (e.g., bell,
`emergency vehicle, security system, etc.), siren (e.g. police
`car, ambulance, etc.), Voice (e.g., "help', 'stop', 'police'.
`etc.), or specific noise type (e.g., breaking glass, gunshot,
`etc.). The memory 208 can store sound signatures for previ
`ously learned target sounds from which the processor 206
`refers to for detecting target Sounds. The Sound signatures can
`be resident in the memory 208 or downloaded to the earpiece
`100 via the transceiver 204 during operation as needed. Upon
`detecting a target Sound, the processor 206 can report the
`target to the user via audio delivered from the ECR 120 to the
`ear canal.
`0029. The earpiece 100 can also include an audio interface
`212 operatively coupled to the processor 206 to receive audio
`content, for example from a media player, and deliver the
`audio content to the processor 206. The processor 206 respon
`sive to detecting target Sounds can adjust the audio content
`and the target Sounds delivered to the earcanal. The processor
`206 can actively monitor the sound exposure level inside the
`ear canal and adjust the audio to within a safe and Subjectively
`optimized listening level range. The processor 206 can utilize
`computing technologies Such as a microprocessor, Applica
`tion Specific Integrated Chip (ASIC), and/or digital signal
`processor (DSP) with associated storage memory 208 such a
`Flash, ROM, RAM, SRAM, DRAM or other like technolo
`gies for controlling operations of the earpiece device 100.
`0030 The earpiece 100 can further include a transceiver
`204 that can Support singly or in combination any number of
`wireless access technologies including without limitation
`BluetoothTM, Wireless Fidelity (WiFi), Worldwide Interoper
`ability for Microwave Access (WiMAX), and/or other short
`or long range communication protocols. The transceiver 204
`can also provide Support for dynamic downloading over-the
`air to the earpiece 100. It should be noted also that next
`generation access technologies can also be applied to the
`present disclosure.
`0031. The power supply 210 can utilize common power
`management technologies such as replaceable batteries, Sup
`ply regulation technologies, and charging system technolo
`gies for Supplying energy to the components of the earpiece
`100 and to facilitate portable applications. A motor (not
`shown) can be a single Supply motor driver coupled to the
`power Supply 210 to improve sensory input via haptic vibra
`tion. As an example, the processor 206 can direct the motor to
`vibrate responsive to an action, such as a detection of a target
`Sound or an incoming Voice call.
`
`Exhibit 1022
`Page 08 of 14
`
`

`

`US 2008/0240458 A1
`
`Oct. 2, 2008
`
`0032. The earpiece 100 can further represent a single
`operational device or a family of devices configured in a
`master-slave arrangement, for example, a mobile device and
`an earpiece. In the latter exemplary embodiment, the compo
`nents of the earpiece 100 can be reused in different form
`factors for the master and slave devices.
`0033 FIG. 3 is a flowchart of a method 300 for earpiece
`monitoring and target detection in accordance with an exem
`plary embodiment. The method 300 can be practiced with
`more or less than the number of steps shown and is not limited
`to the order shown. To describe the method 300, reference
`will be made to components of FIG. 2, although it is under
`stood that the method 300 can be implemented in any other
`manner using other suitable components. The method 300
`can be implemented in a single earpiece, a pair of earpieces,
`headphones, or other suitable headset audio delivery device.
`0034. The method 300 can start in a state wherein the
`earpiece 100 has been inserted and powered on. As shown in
`step 302, the processor 206 can monitor the environment for
`target Sounds, such as an alarm, a horn, a voice, or a noise.
`Each of the target sounds can have certain identifiable fea
`tures that characterize the sound. The features can be collec
`tively referred to as a sound signature which can be used for
`recognizing the target Sound. As an example, the Sound sig
`nature may include statistical properties or parametric prop
`erties of the target Sound. For example, a Sound signature can
`describe prominent frequencies with associated amplitude
`and phase information. As another example, the sound signa
`ture can contain principal components identifying the most
`likely recognizable features of a target Sound.
`0035. The processor 206 at step 304 can then detect the
`target Sounds within the environment based on the Sound
`signatures. As will be shown ahead, feature extraction tech
`niques are applied to the ambient sound captured at the ASM
`110 to generate the Sound signatures. Pattern recognition
`approaches are applied based on known Sound signatures to
`detect the target Sounds from their corresponding Sound sig
`natures. More specifically, Sound signatures can then be com
`pared to learned models to identify a corresponding target
`Sound. Notably, the processor 206 can detect sound signa
`tures from the ambient sound regardless of the state of the
`earpiece 100. For example, the earpiece 100 may be in a
`listening state wherein ambient Sound is transparently passed
`to the ECR 120, in a media state wherein audio content is
`delivered from the audio interface 212 to the ECR 120, or in
`an active listening state wherein Sounds in the environment
`are selectively enhanced or suppressed.
`0036. At step 306, the processor 206 can adjust sound
`delivered to the ear canal in view of a detected target sound.
`For instance, if the earpiece is in a listening state, the proces
`Sor 206 can amplify detected target Sounds inaccordance with
`a Personalized Hearing Level (PHL). The PHL establishes
`comfortable and uncomfortable levels of hearing, and can be
`referenced by the processor 206 to set the volume level of the
`target Sound (or ambient sound) so as not to exceed the user's
`preferred listening levels. As another example, if the earpiece
`is in a media State, the processor 206 can attenuate the audio
`content delivered to the ear canal, and amplify the target
`sounds in the ear canal. The PHL can also be used to properly
`mix the volumes of the different sounds. As yet another
`example, if the earpiece 100 is in an active state, the processor
`206 can selectively adjust the volume of the target sounds
`relative to background noises in the environment.
`
`0037. The processor 206 can also compensate for an ear
`seal leakage due to a fitting of the earpiece 100 with the ear
`canal. An ear seal profile can be generated by evaluating
`amplitude and phase difference between the ASM 110 and the
`ECM 202 for known signals produced by the ECR 120. That
`is, the processor 120 can monitor and report transmission
`levels offrequencies through the earcanal 140. The processor
`206 can take into account the ear seal leakage when perform
`ing audio enhancement, or other spectral enhancement tech
`niques, to maintain minimal audibility of the ambient noise
`while audio content is playing.
`0038. Upon detecting a target sound in the ambient sound
`of the user's environment, the processor at step 308 can gen
`erate an audible alarm within the ear canal that identifies the
`detected Sound signature. The audible alarm can be a repro
`duction of the target Sound, an amplification of the target
`Sound (or the entire ambient Sound), a text-to-speech message
`(e.g. synthetic Voice) identifying the target Sound, a haptic
`vibration via a motor in the earpiece 100, oran audio clip. For
`example, the earpiece 100 can play a Sound bite (i.e., audio
`clip) corresponding to the detected target Sound such as an
`ambulance, fire engine, or other environmental sound. As
`another example, the processor 206 can synthesize a Voice to
`describe the detected target Sound (e.g., “ambulance
`approaching”).
`0039 FIG. 4 illustrates earpiece modes in accordance with
`an exemplary embodiment. The earpiece mode can be manu
`ally selected by the user, for example, by pressing a button, or
`automatically selected, for example, when the earpiece 100
`detects it is in an active listen state or in a media state. As
`shown in FIG. 4, the earpiece mode can correspond to Sig
`nature Sound Pass Through Mode (SSPTM), Signature
`Sound Boost Mode (SSBM), Signature Sound Replacement
`Mode (SSRM), Signature Sound Attenuation Mode (SSAM),
`and Signature Sound Replacement Mode (SSRM).
`0040. In SSPTM mode, ambient sound captured at the
`ASM 110 is passed transparently to the ECR 120 for repro
`duction within the ear canal. In this mode, the Sound produced
`in the ear canal Sufficiently matches the ambient sound out
`side the ear canal, thereby providing a “transparency effect.
`That is, the earpiece 100 recreates the sound captured at the
`ASM 110 to overcome occlusion effects of the earpiece 100
`when inserted within the ear. The processor 206 by way of
`sound measured at the ECM 130 adjusts the properties of
`sound delivered to the ear canal so the sound within the
`occluded ear canal is the same as the ambient sound outside
`the ear, as though the earpiece 100 were absent in the ear
`canal. In one configuration, the processor 206 can predict an
`approximation of an equalizing filter to provide the transpar
`ency by comparing an ASM 110 signal and an ECM 130
`signal transfer function.
`0041. In SSBM, target sounds and/or ambient sounds are
`amplified upon the processor 206 detecting a target Sound.
`The target sound can be amplified relative to the normal level
`received, or amplified above an audio content level if audio
`content is being delivered to the ear canal. As noted previ
`ously, the target Sound can also be amplified in accordance
`with a user's PHL to be within safe hearing levels, and within
`subjectively determined listening levels.
`0042. In SSRM, target sounds detected in the environment
`can be replaced with audible warning messages. For example,
`the processor 206 upon detecting a target Sound can generate
`synthetic speech identifying the target Sound (e.g., “ambu
`lance detected'). In such regard, the earpiece 100 audibly
`
`Exhibit 1022
`Page 09 of 14
`
`

`

`US 2008/0240458 A1
`
`Oct. 2, 2008
`
`reports the target sound identified thereby relieving the user
`from having to interpret the target Sound. The synthetic
`speech can be mixed with the ambient sound (e.g., amplified,
`attenuated, cropped, etc.), or played alone with the ambient
`Sound muted.
`0043. In SSAM, sounds other than target sounds can be
`attenuated. For instance, annoying Sounds or noises not asso
`ciated with target Sounds can be Suppressed. For instance, by
`way of a learning session, the user can establish what sounds
`are considered target Sounds (e.g., “ambulance') and which
`Sounds are non-target Sounds (e.g. jackhammer”). The pro
`cessor 206 upon detecting non-target Sounds can thus attenu
`ate these sounds within the occluded or partially occluded ear
`canal.
`0044 FIG. 5 is a flowchart of a method 500 for a method
`for Sound signature detection in accordance with an exem
`plary embodiment. The method 500 can be practiced with
`more or less than the number of steps shown and is not limited
`to the order shown. To describe the method 500, reference
`will be made to components of FIG. 2, although it is under
`stood that the method 500 can be implemented in any other
`manner using other suitable components. The method 500
`can be implemented in a single earpiece, a pair of earpieces,
`headphones, or other suitable headset audio delivery device.
`0045. The method can start at step 502, in which the ear
`piece 100 can enter a learn mode. Notably, the earpiece upon
`completion of a learning mode or previous learning configu
`ration can start instead at step 520. In the learning mode of
`step 502, the earpiece 100 can actively generate and learn
`Sound signatures from ambient Sounds within the environ
`ment. In learning mode, the earpiece 100 can also receive
`previously trained learning models to use for detecting target
`Sounds in the environment. In an active learning mode, the
`user can press a button or otherwise (e.g. Voice recognition)
`initiate a recording of ambient Sounds in the environment. For
`example, the user can upon hearing a new target Sound in the
`environment ("car horn'), activate the earpiece 100 to learn
`the new target Sound. Upon generating a sound signature for
`the new target Sound, it can be stored in the user defined
`database 504. In another arrangement, the earpiece 100 upon
`detecting a unique Sound, characteristic to a target Sound, can
`ask the user if they desire to have the sound signature for the
`unique Sound learned. In Such regard, the earpiece 100
`actively senses sounds and queries the user about their envi
`ronment to learn the Sounds. Moreover, the earpiece can
`organize learned sounds based on environmental context, for
`example, in outdoor (e.g. traffic, car, etc.) or indoor (e.g.,
`restaurant, airport) environments.
`0046. In another learning mode, trained models can be
`retrieved from an on-line database 506 for use in detecting
`target Sounds. The previously learned models can be trans
`mitted on a scheduled basis to the earpiece, or as needed,
`depending on the environmental context. For example, upon
`the earpiece 100 detecting traffic noise, Sound signature mod
`els associated with target Sounds (e.g., ambulance, police car)
`in traffic can be retrieved. In another exemplary embodiment,
`upon the earpiece 100 detecting conversational noise (e.g.
`people talking), Sound signature models for Verbal warnings
`(“help”, “police') can be retrieved. Groups ofsound signature
`models can be retrieved based on the environmental context
`or on user directed action.
`0047. As shown in step 508, the earpiece can also generate
`speech recognition models for target Sounds corresponding to
`voice, such as “help', 'police”, “fire', etc. The speech recog
`
`nition models can be retrieved from the on-line database 506
`or the user defined database 504. In the latter for example, the
`user can say a word or enter a text version of a word to
`associate with a verbal warning Sound. For instance, the user
`can define a set of words of interest along with mappings to
`their meanings, and then use keyword spotting to detect their
`occurrences. If the user enters an environment wherein
`another individual says the same word (e.g., “help') the ear
`piece 100 can inform the user of the verbal warning sound.
`For other acoustic sounds, the earpiece 100 can generate
`sound signature models as shown in step 510. Notably, the
`earpiece 100 itself can generate the Sound signature models,
`or transmit the captured target Sounds to external systems
`(e.g., remote server) that generate the sound signature mod
`els. Such learning can be conducted off-line in a training
`phase, and the earpiece 100 can be uploaded with the new
`learning models.
`0048. It should also be noted that the learning models can
`be updated during use of the earpiece, for example, when the
`earpiece 100 detects target sounds. The detected target sounds
`can be used to adapt the learning models as new target Sound
`variants are encountered. For example, the earpiece 100 upon
`detecting a target Sound, can use the Sound signature of the
`target Sound to update the learned models in accordance with
`the training phase. In Such an exemplary embodiment a first
`learned model is adapted based on new training data collected
`in the environment by the earpiece. In Such regard, for
`example, a new set of “horn’ target sounds could be included
`in real-time training without discarding the other "horn'
`Sounds already captured in the existing model.
`0049. Upon completion of learning, uploading, or
`retrieval of sound signature models, the earpiece 100 can
`monitor and report target Sounds within the environment. As
`shown in step 520, ambient Sounds (e.g. input signal) within
`the environment are captured by the ASM 110. The ambient
`sounds can be digitized by way of the ADC 202 and stored
`temporarily to a data buffer in memory 208 as shown in step
`522. The data buffer holds enough data to allow for generation
`of a sound signature as will be described ahead in FIG. 7.
`0050. In another configuration, the processor 206 can
`implement a “lookahead' analysis system by way of the data
`buffer for reproduction of pre-recorded audio content, using a
`data buffer to offset the reproduction of the audio signal. The
`look-ahead system allows the processor to analyze poten
`tially harmful audio artifacts (e.g. high level onsets, bursts,
`etc.) either received from an external media device, or
`detected with the ambient microphones,