US008837257B2
`
`(12) United States Patent
`Surprenant et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,837,257 B2
`Sep. 16, 2014
`
`(54) ACOUSTIC MODULATION PROTOCOL
`
`7,796,978 B2
`2002/0101917 A1
`
`9/2010 Jones et al.
`8, 2002 Bibl
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. 381.60
`
`28393.5. A. * 1 58 Sansky .
`2003. O144035 A1
`7, 2003 Weinblatt
`2003/0232593 A1 12/2003 Wahroos et al.
`2004/0031856 A1
`2/2004 Atsmon
`2005/0223030 A1 10, 2005 Morris
`(Continued)
`
`(75) Inventors: Richard S. Surprenant, Santa Clara,
`CA (US); Chad G. Seguin, Morgan Hill,
`CA (US); Brett L. Paulson, Palo Alto,
`CA (US)
`
`(73) Assignee: Verifone Systems, Incorporated, San
`Jose, CA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 519 days.
`
`(*) Notice:
`
`(21) Appl. No.: 13/151,516
`
`(22) Filed:
`(65)
`
`Jun. 2, 2011
`Prior Publication Data
`US 2012/0134238A1
`May 31, 2012
`O
`O
`Related U.S. Application Data
`(60) Provisional application No. 61/417,705, filed on Nov.
`29, 2010.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(51) Int. Cl.
`H04B I/O
`HD4L27/00
`H04L 25/38
`HO4S I/O
`(52) U.S. Cl.
`CPC ...................................... H04S I/007 (2013.01)
`USPC ............ 367/137; 375/141; 375/370; 375/326
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,359,367 A 10, 1994 Stock11
`5,604,771 A * 2/1997 Quiros .......................... 375/326
`6,038.436 A
`3, 2000 Priest
`
`FOREIGN PATENT DOCUMENTS
`
`GB
`GB
`
`2363275 A 12/2001
`2363275 B
`T 2002
`OTHER PUBLICATIONS
`
`Nielsen Media-Sync Platform Backs ABC's My Generation Sync
`iPad App WebNewser, David Cohen, Sep. 16, 2010.
`(Continued)
`Primary Examiner — Isam Alsomiri
`Assistant Examiner — Hovhannes Baghdasaryan
`(74) Attorney, Agent, or Firm — Convergent Law Group
`LLP
`ABSTRACT
`(57)
`Exemplary embodiments provide a computer-implemented
`method for generating a modulated acoustic carrier signal for
`wireless transmission from a speaker of a transmit device to a
`microphone of a receive device. Aspects of the exemplary
`embodiments include converting a message to binary data;
`modulating one or more selected frequencies for one or more
`acoustic carrier signals based on the binary data to generate
`one or more modulated acoustic carrier signals; filtering the
`one or more modulated acoustic carrier signals to remove any
`unintended audible harmonics created during modulation,
`including; equalizing the modulated acoustic carrier signal to
`pre-compensate for known degradations that will occur fur
`ther along a signal path; setting a level of the modulated
`acoustic carrier signal for the intended application; and stor
`ing the modulated acoustic carrier signal in a buffer for Sub
`sequent output and transmission by the speaker.
`
`25 Claims, 16 Drawing Sheets
`
`2
`
`Eran
`Gair
`3ga
`
`Rastaries:
`Digital Fourier
`Transfort trf
`96.
`
`Bt
`Restrusion
`908
`
`
`
`and
`
`ite
`
`Peakeweletector
`
`stgahso peak is Less that
`the detected peak leveland to
`grace the signal to approx.
`o.9 of detected peakewei
`at
`918
`
`
`
`Seertzlfrequency Adjust Window
`Transform
`/ 92
`920
`
`Level matching
`322
`
`Subbit Averaging
`924
`
`igital data
`925
`
`Demaduation Signal Conditioning
`
`Page 1 of 28
`
`GOOGLE EXHIBIT 1006
`
`

`

`US 8,837,257 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2005.0246747 A1 11, 2005 Braun et al.
`2007/0211786 A1* 9, 2007 Shattil ........................... 375,141
`2008, OO39135 A1
`2/2008 Nakamura et al.
`2008/0273644 A1* 11/2008 Chesnutt et al. .............. 375/370
`2008/0306736 A1 12/2008 Sanyal
`2009/0041260 A1
`2/2009 Jorgensen
`2009,0059903 A1
`3, 2009 Kubler
`2009/0076825 A1
`3, 2009 Bradford
`2009, O154603 A1
`6, 2009 Li
`2009/0156193 A1
`6/2009 Urbania
`
`7, 2009 Knudsen
`2009, O169038 A1
`2009, O305,677 A1 12/2009 Ellison et al.
`2010, 0119208 A1
`5, 2010 Davis et al.
`2012fOO787O2 A1
`3/2012 Rissanen
`2013/0010979 A1
`1/2013 Takara
`
`OTHER PUBLICATIONS
`
`MetaMirror & The Future of TV. http://www.designbynotion.com/
`metamirror-next-generation-tv/, Aug. 5th, 2010.
`Mozaik, http://mymosaic.net/www/icontent.php.
`
`* cited by examiner
`
`Page 2 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 1 of 16
`
`US 8,837,257 B2
`
`Receive Device 103
`
`
`
`100
`
`
`
`MOculated ACOustic
`-- Carrier Signal
`111
`
`Transmit Device 101
`
`102
`
`MEMORY
`ACoustic Communication
`Modulation Component
`Acoustic Communication
`Demodulation Component
`ACOustic Transmission
`Strategy Component
`Run-time Environment
`
`126
`. 124
`
`122
`
`120
`104
`
`/
`
`Sound
`Components
`
`Processor
`Complex
`
`Broadband
`Interface
`
`Data/Voice
`Interface
`
`FIG. 1
`
`Page 3 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 2 of 16
`
`US 8,837,257 B2
`
`5000
`
`Level (dBFS)
`10000
`15000
`
`20000
`
`- Level (dBFS)
`
`Music and
`Mechanicat Noi
`
`FIG. 2A
`
`Recorded Data Signal over Air Interface
`5000
`10000
`15OOO
`2OOOO
`
`
`
`Electrical Noise
`(ex. Motor or Ballast)
`
`NY/
`-70 "awe- I
`\"", W
`-80
`
`h.
`
`Spectrum Level
`(dBFS)
`
`FIG. 2B
`
`Page 4 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 3 of 16
`
`US 8,837,257 B2
`
`esuodsay! —
`
`
`
`
`
`
`
`
`
`000ZZ 0000Z 0008), 0009), 000$'}, 000Z? 0000},000800090007000Z0
`
`
`
`
`
`
`
`S-9peNe
`
`Page 5 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 4 of 16
`
`US 8,837,257 B2
`
`300 - Message Data
`
`
`
`302 -- ( Mae
`
`304 -
`y
`
`C Equalize D
`
`308 -
`
`\
`\
`\
`
`Encoding Process
`
`Band Pass
`— Filter
`
`Locking
`Segment
`4OO
`
`Mark
`Segment
`402
`
`Data
`Segment
`404
`
`Modulated Acoustic Carrier Signal 111
`
`FIG. 4
`
`Page 6 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 5 of 16
`
`| | | | | | ||
`| WW
`IV WWII
`
`FIG. 5
`
`Page 7 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 6 of 16
`
`US 8,837,257 B2
`
`
`
`
`
`
`
`3SION pueg O!pnw
`
`
`
`
`
`
`
`
`
`
`
`OOOZZ 0000Z 0008) 0009), 0007), 000Z? 0000\ 0008 0009 0007 000 ZZH
`
`
`
`
`
`
`
`
`
`
`
`Page 8 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 7 of 16
`
`US 8,837,257 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`000ZZ 0000Z 0008), 0009), 0007), 000Z) 0000? 0008 0009 0007 000ZZH
`
`
`
`
`
`
`
`
`
`
`
`Page 9 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 8 of 16
`
`US 8,837,257 B2
`
`an
`9.
`s
`O
`2
`s
`s
`
`
`
`an
`9.
`CD
`O
`2
`E
`d
`s
`D
`
`O
`
`5
`
`20
`
`25
`
`15
`10
`Frequency (Hz)
`FIG. 8A
`
`O
`
`5
`
`20
`
`25
`
`15
`10
`Frequency (Hz)
`FIG. 8B
`
`Page 10 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 9 of 16
`
`US 8,837,257 B2
`
`
`
`
`
`
`
`g
`g
`
`2
`
`-60 NAAAA
`
`YYYYY
`
`-80
`
`-100
`
`XLY
`
`y A
`
`YYYYYY
`
`
`
`20
`
`25
`
`-120
`O
`
`5
`
`15
`10
`Frequency (Hz)
`FIG. 8C
`
`
`
`g
`g
`2
`C g
`d
`
`O
`
`5
`
`15
`10
`Frequency (Hz)
`FIG. 8D
`
`20
`
`25
`
`Page 11 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`US 8,837,257 B2
`
`q=-
`S)
`
`y T /Y
`
`©
`
`Q
`
`==+----
`
`)
`
`
`
`20
`
`25
`
`5 1
`Frequency (Hz)
`FIG. 8E
`
`Frequency (Hz)
`FIG. 8F
`
`Page 12 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 11 of 16
`
`US 8,837,257 B2
`
`
`
`Frequency (Hz)
`FIG. 8G
`
`Frequency (Hz)
`FIG. 8H
`
`Page 13 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 12 of 16
`
`US 8,837,257 B2
`
`Block
`Processing
`900
`
`Filtering Y
`902
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Audio
`Stfear
`910
`
`Block of Audio Data
`
`
`
`Bandpass filter
`914
`
`Peak Levet Detector
`916
`
`Adjust gain so peak is Less than
`the detected peak level and to
`normalize the signal to approx.
`0.7 to 9 of detected peak level
`
`918
`
`X
`
`Rasterized
`Digital Fourier
`Transform (RFT)
`906
`y
`
`
`
`
`
`
`
`
`
`f
`
`
`
`Bit
`se
`Reconstruction
`
`Goertz Frequency Adjust Window
`Transform
`w-1 921
`
`Level matching
`922.
`
`sub bit Averaging
`924
`
`
`
`?digital dataY
`926
`
`Demodulation Signal Conditioning
`
`FIG. 9
`
`Page 14 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 13 of 16
`
`US 8,837,257 B2
`
`3
`
`9
`
`
`
`Time (mS)
`
`Time (mS)
`
`FIG. 1 OB
`
`
`
`
`
`
`
`Page 15 of 28
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 16, 2014
`Sep. 16, 2014
`
`Sheet 14 of 16
`Sheet 14 of 16
`
`US 8,837,257 B2
`US 8,837,257 B2
`
`B
`
`ykn=F(uk)
`
`Y« INJ=F( wd)
`
`
`
`xn)
`x{n]
`
`Page 16 of 28
`
`FIG. 11
`FIG. 11
`
`Page 16 of 28
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 16, 2014
`Sep. 16, 2014
`
`Sheet 15 of 16
`Sheet 15 of 16
`
`US 8,837,257 B2
`US 8,837,257 B2
`
`-
`will
`5
`3
`OD
`O
`¢
`s
`=
`an
`ao
`
`N
`CN
`s
`3
`Oo
`O
`.
`&
`3.
`=
`CS
`Oo
`
`wo
`cr)
`s
`z
`wo
`O
`©
`s
`=
`a
`
`50
`
`
`
`200250
`
`3
`
`150
`
`3
`100
`
`A.AudioDataBuffer
`
`Page 17 of 28
`
`FIG.12
`
`Page 17 of 28
`
`

`

`U.S. Patent
`
`Sep. 16, 2014
`
`Sheet 16 of 16
`
`US 8,837,257 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Buffer 1300
`
`< 30
`frequency
`transitions
`
`30
`transitions
`received
`
`Wait for
`Start
`1304
`
`Start bit
`received
`
`
`
`
`
`Data
`incomplete
`
`**TimeOut"
`
`
`
`Checksum
`fails
`
`
`
`Reset States
`1312
`
`Receive
`Data
`1306
`
`Data
`complete
`
`
`
`Verify checksum
`1308
`
`Checksum
`Succeeds
`
`Send data event
`1310
`
`DeCOder State Model
`
`FIG. 13
`
`Page 18 of 28
`
`

`

`US 8,837,257 B2
`
`1.
`ACOUSTIC MODULATION PROTOCOL
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims benefit of priority to Provisional
`Patent Application 61/417,705 filed Nov. 29, 2010, and is
`related to patent application Ser. No. 12/870,767, filed Aug.
`27, 2010, both assigned to the assignee of the present appli
`cation, and incorporated herein by reference.
`
`10
`
`BACKGROUND
`
`2
`inserted into audio at points where they are imperceptible. TV
`networks and broadcasters use equipment called a NAVE
`(Nielsen Audio Video Encoder) to “burp” these signature
`codes into the program audio, which are picked up by devices
`installed in 40,000 viewer's homes—the company’s statisti
`cal sample base. Called NP (Active/Passive) monitors, these
`cable-box-sized gadgets tie into the audio output of a TV or
`home theater system and actively decode and store the psy
`choacoustic signals.
`Despite these advances in data transmission via audio,
`current technology has limitations that hamper widespread
`adoption. For example Spot411 works only with Blu-ray or
`DVDs (not broadcasts) and must use an audio signature of a
`movie to identify which movies is being played. The app must
`also first be synched with the movie.
`Shopkick requires additional hardware—a separate
`speaker, to produce the acoustic signal, and determine the
`presence of the device running the Shopkick application.
`There is no transmission of any other audio signal besides the
`inaudible Shopkick signal.
`The Neilson system is used to determine when a show is
`really viewed, vs. the time of its scheduled broadcast; the
`codes circumvent the problem of time-shifted viewing,
`because the audio burps also show up on recorded programs
`when played from a DVR hard drive or VCR tape. The sig
`natures, however, are only used to ID broadcast programs
`from which the signatures were derived.
`In addition, many prior art solutions are based on fre
`quency shift keying (FSK) for modulation/demodulation,
`which has had limitations in an acoustic communications
`environment.
`Accordingly, it would be desirable to provide an improved
`over air acoustic data communication method and system.
`
`BRIEF SUMMARY
`
`The exemplary embodiments provide a computer-imple
`mented method for generating a modulated acoustic carrier
`signal for wireless transmission from a speaker of a transmit
`device to a microphone of a receive device. Aspects of the
`exemplary embodiments include converting a message to
`binary data; modulating one or more selected frequencies for
`one or more acoustic carrier signals based on the binary data
`to generate one or more modulated acoustic carrier signals;
`filtering the one or more modulated acoustic carrier signals to
`remove any unintended audible harmonics created during
`modulation, including; equalizing the modulated acoustic
`carrier signal to pre-compensate for known degradations that
`will occur further along a signal path; setting a level of the
`modulated acoustic carrier signal for the intended applica
`tion; and storing the modulated acoustic carrier signal in a
`buffer for Subsequent output and transmission by the speaker.
`
`BRIEF DESCRIPTION OF SEVERAL VIEWS OF
`THE DRAWINGS
`
`FIG. 1 is a block diagram illustrating an exemplary acous
`tics system in which the acoustic modulation protocol may be
`implemented.
`FIG. 2A is a graph illustrating a typical noise spectrum as
`measured by the receive device microphone; FIG. 2B is a
`graph illustrating levels of the modulated acoustic carrier
`signal wirelessly received over the air at various frequencies
`Versus noise levels; and FIG. 2C is a graph depicting an
`end-to-end system response.
`
`15
`
`25
`
`30
`
`35
`
`40
`
`Over the years, technology has been developed for trans
`mitting data over the air using an audio signal from a speaker.
`For example, Spot411 Technologies provides an iPhone
`application (app) for NBC Universal and 20th Century Fox
`that tap into DVD or Blu-ray discs to augment viewing. The
`app uses the microphone on the iPhone or a laptop, to “hear
`the audio signal from the movie being played, then responds
`with pop-ups about the movie. It also intersects with Face
`book and Twitter for movie chats. The app makes the DVD
`part of a networked experience. Universal's Pocket Blu app is
`just for the iPhone and enables the iphone to act as a remote
`control for the movie if when played on a Blu-ray player (it
`currently doesn’t work with traditional DVDs or computers)
`and plays trailers for upcoming movies. Using the internal
`microphones of the device, 20th Century Fox's FoxPop app
`listens and synchs up with the film, and then delivers random
`facts, trivia and behind-the-scenes details that pop up for the
`viewer at specific points throughout the film. The app can let
`auser leave a message for a friend who mightwatch the movie
`in the future.
`Another example is ShopKick’s iPhone application that
`takes advantage of a Smartphone's microphone to bring loca
`tion-sensing indoors, where GPS won't work, for location
`based shopping. Beacons Smaller than a person’s hand fixed
`to a store's ceiling beam out an inaudible ultrasound signal at
`a frequency that can be picked up by a cell phone's micro
`phone but not by human ears. The app decodes the signal and
`contacts ShopKick’s database to determine where the user is,
`and to retrieve some sort of reward for the user.
`Neilson offers a service where data transmission via audio
`signals is used for video on demand reporting. A video on
`demand (VOD) Audience Measurement service enables con
`45
`tent providers to insert a digital audio watermark into VOD
`content which is audible to Nielsen meters in homes.
`Neilson also has personal meters, called “Go Meters,” that
`capture out-of-home viewing by collecting audio signatures.
`One device places metering technology in cellphones and the
`other is a customized meter that resembles an MP3 player.
`The Meters recognize when a show is playing, based on
`signals hidden in the audio. One method, called psychoacous
`tic encoding, injects a digital time stamp and program title—
`or “active signature' into the audio tracks of TV shows as
`they are broadcast. Another technique, called passive signa
`tures, creates a kind of audio fingerprint for TV shows; a
`split-second sample of audio is digitized, creating a unique
`signature, which also can be recognized by metering equip
`ment.
`The psychoacoustic encoding method relies on digital sig
`nals embedded in the audio of broadcast TV shows. These
`signals—which last for a fraction of a second are slipped
`into the audio tracks of TV shows approximately every 2.5
`seconds, except for periods of Sustained silence. If heard, the
`encoded signals would be a crrrkkkkk kind of sound. While
`the codes themselves can be heard by the human ear, they are
`
`50
`
`55
`
`60
`
`65
`
`Page 19 of 28
`
`

`

`US 8,837,257 B2
`
`3
`FIG. 3 is flow diagram illustrating a process for generating
`a modulated acoustic carrier signal for wireless transmission
`from a speaker of a transmit device to a microphone of a
`receive device.
`FIG. 4 is a block diagram illustrating components of the
`modulated acoustic carrier signal according the AMP.
`FIG. 5 is a graph illustrating an example of a modulated
`acoustic carrier signal having phase coherent transitions.
`FIG. 6 is a graph showing frequency spectrum and har
`monic artifacts of the modulated acoustic carrier signal
`before filtering.
`FIG. 7 is a graph showing the frequency spectrum of the
`modulated acoustic carrier signal after filtering.
`FIGS. 8A-8H are graphs illustrating example filter
`responses.
`FIG.9 is a block diagram illustrating a process for demodu
`lating the modulated acoustic carrier signal according to one
`embodiment.
`FIG. 10A is a graph showing an energy envelope of the
`received modulated acoustic carrier signal; and FIG. 10B is a
`graph showing the energy envelope of modulated acoustic
`carrier signal after the dynamic gain stage.
`FIG.11 is a block diagram showing an example implemen
`tation of a Goertzel Algorithm.
`FIG. 12 is a diagram illustrating examples of three over
`lapping Goertzel analysis windows.
`FIG. 13 is a block diagram illustrating details of the
`decoder state machine to lock onto the modulated acoustic
`carrier signal.
`
`DETAILED DESCRIPTION
`
`4
`and decoded to recover the digital data using the acoustic
`modulation protocol. The resulting acoustic carrier signal
`with the encoded digital data Supports numerous bitrates. The
`acoustic modulation protocol can be used for unidirectional
`orbidirectional data communication.
`The acoustics modulation protocol of the exemplary
`embodiments enable two or more devices to communicate
`acoustically with one another without the need for specialized
`hardware (e.g., near field communication (NFC), global posi
`tioning system (GPS), Bluetooth (BT), chips, RFID tags,
`dongles, and the like) other than microphones and speakers
`found on most computers and portable devices. In one
`embodiment, the modulation techniques of the acoustics
`modulation protocol can be applied using any carrier fre
`quency value, some of which may be audible (<20 kHz) or
`inaudible (>20 kHz). The choice of which carrier frequency
`to use may be dependent upon the user application, ambient
`noise conditions, and frequency response of the acoustic sys
`tem.
`FIG. 1 is a block diagram illustrating an exemplary acous
`tics system in which the acoustic modulation protocol may be
`implemented. The acoustics system 100 includes a transmit
`device 101 and a receive device 103. Transmit device 101
`may include a memory 102, sound components 104 with
`speaker 106 and/or microphone 108, a processor complex
`110, a broadband interface 112, data/voice interface 114 and
`system storage 116. Receive device 103 may include the same
`components, but the speaker may be optional. In one embodi
`ment, one or both of the transmit device 101 and the receive
`device 103 may apply to any type of wireless phone, com
`puter enabled devices (i.e., point-of-sale terminals, electronic
`billboards, kiosks) or general-purpose computers capable of
`performing acoustic communication in accordance with the
`present invention. To that end, transmit device 101 may also
`be broadly, and alternatively, referred to as a mobile device,
`wireless phone, Smartphone, feature phone, computer, laptop
`computer, tablet or Smartbook. Moreover, various aspects of
`the invention may include the same or similar components
`despite the particular implementation illustrated in FIG. 1.
`For example, some implementations may use a central inter
`connect 118 for communication among the components
`while other implementations may use multiple direct paths
`between each of the components. Alternate embodiments
`may combine one or more of these components into a single
`component or may separate them into different combinations
`of components. Functionality provided by the transmit device
`101 and receive device 103 may be implemented in hardware,
`Software or in various combinations thereof depending on the
`design and implementation details.
`In the illustrative implementation in FIG. 1, memory 102
`includes storage locations that are addressable by the proces
`Sor complex 110 and adapters for storing software program
`code and data. For example, memory 102 may comprise a
`form of random access memory (RAM) that is generally
`classified as “volatile' memory. Processor complex 110 and
`various adapters may, in turn, comprise processing elements
`and logic circuitry configured to execute the software code
`and manipulate the data stored in the memory 102. System
`storage 116 may be a form of non-volatile storage for storing
`a copy of run-time environment 120, applications and other
`data used by transmit device 101.
`According to the exemplary embodiment, the transmit
`device 101 is enabled with an acoustic modulation protocol
`(AMP) 107. The acoustic modulation protocol (AMP) 107
`may reside in memory 102 during run-time and may include
`an acoustic communication modulation component 126, an
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`The exemplary embodiment relates to an acoustic modu
`lation protocol. The following description is presented to
`enable one of ordinary skill in the art to make and use the
`invention and is provided in the context of a patent application
`and its requirements. Various modifications to the exemplary
`embodiments and the generic principles and features
`described herein will be readily apparent. The exemplary
`embodiments are mainly described in terms of particular
`methods and systems provided in particular implementations.
`However, the methods and systems will operate effectively in
`other implementations. Phrases such as “exemplary embodi
`ment”, “one embodiment” and “another embodiment may
`refer to the same or different embodiments. The embodiments
`45
`will be described with respect to systems and/or devices
`having certain components. However, the systems and/or
`devices may include more or less components than those
`shown, and variations in the arrangement and type of the
`components may be made without departing from the scope
`of the invention. The exemplary embodiments will also be
`described in the context of particular methods having certain
`steps. However, the method and system operate effectively
`for other methods having different and/or additional steps and
`steps in different orders that are not inconsistent with the
`exemplary embodiments. Thus, the present invention is not
`intended to be limited to the embodiments shown, but is to be
`accorded the widest scope consistent with the principles and
`features described herein.
`The exemplary embodiments provide an acoustic modula
`tion protocol (AMP) for enabling transmission of a signal
`over an acoustics interface. The exemplary embodiments take
`advantage of existing Sound components of a mobile device,
`Such as a speaker, to encode digital data on an acoustic carrier
`signal using an acoustic modulation protocol. The acoustic
`carrier signal is sent over air where it is received by existing
`Sound components of a receive device. Such as a microphone,
`
`Page 20 of 28
`
`

`

`5
`acoustic communication demodulation component 124, and
`an acoustic transmission strategy component 122.
`Acoustic communication modulation component 126
`includes functions and datasets that encode data and modu
`late it over acoustic transmission frequencies, creating a
`modulated acoustic carrier signal 111 in accordance with the
`exemplary embodiment. Likewise, acoustic communication
`demodulation component 124 includes functions and
`datasets necessary to demodulate data from modulated acous
`tic carrier signals 111 received over various acoustic trans
`mission frequencies in accordance with AMP. Acoustic trans
`mission strategy component 122 includes functions and
`datasets necessary for identifying the acoustic transmission
`frequencies and timing to transmit and receive data acousti
`cally in accordance with aspects of the present invention. For
`example, acoustic transmission strategy component 122 may
`identify the acoustic frequencies for transmitting data and to
`determine an optimal time for acoustically transmitting the
`data. The receive device 103 may include the same AMP
`components, with the exception of the acoustic transmission
`strategy component 122 in an embodiment where receive
`device 103 itself does not transmit a modulated acoustic
`carrier signal 111.
`Memory 102 may also include run-time environment 120
`portions of which typically reside in memory and are
`executed by the processing elements. Run-time environment
`120 may be based upon a general-purpose operating system,
`such as Linux, UNIX(R) or Windows.(R), the Apple OSR or any
`other general-purpose operating system. It may also be based
`upon more specialized operating systems such as the Black
`berry Operating system from RIM, Inc., the Symbian OS
`from Nokia, Inc., the iPhone OS or iOS from Apple, Inc., the
`Android operating system from Google, Inc. of Mountain
`View Calif., the WebOS or HPWebOS from Hewlett Packard
`Co. or any other operating system designed for the mobile
`market place.
`Sound components 104 include codecs and other compo
`nents for converting sound transmitted through microphone
`108 into a digital format such as PCM (pulsecode modula
`tion). These codecs are also capable of converting the digital
`information back into an acoustic analog signal and then
`broadcasting through speaker 106.
`Processor complex 110 may be a single processor, multiple
`processors or multiple processor cores on a single die. It is
`contemplated that processor complex 110 represents the one
`or more computational units available in transmit device 101.
`Processor complex 110 may also be a physical aggregation of
`multiple individual processors that each individually process
`and transfer data over interconnect 118. Alternate implemen
`tations of processor complex 110 may be a single processor
`having multiple on-chip cores that may partition and share
`certain resources also on the processor die Such as L1 L2
`cache. For at least these reasons, aspects of the exemplary
`embodiment may be described as using a processor or mul
`tiple processors for convenience, however, it is contemplated
`that the term “processor could also be applied to designs
`utilizing one core or multiple cores found on a single chip or
`die. Likewise, the term process is used to describe the act of
`executing a set of related instructions on one or several pro
`cessors but it is also contemplated that alternate implementa
`tions could be performed using single or multiple threads
`executing the same or similar instructions on one or several
`processors each capable of multi-threaded execution.
`Broadband interface 112 may be a WiFi, WiMAX or other
`connection to a network such as the Internet. The broadband
`interface 112 may also include wired connections to the Inter
`net using CAT 5/6, Fiber Channel or similar methods. Data/
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,837,257 B2
`
`10
`
`15
`
`6
`Voice interface 114 includes functions and datasets for trans
`mitting data and voice over a wireless network. Protocols
`used for data/voice interface 114 may include one or more of
`GSM, CDMA, TDMA, FDMA or other wireless protocols.
`The data portions of data/voice interface 114 may carry data
`at 2G, 2.5G, 3G, 4G and beyond implemented using various
`wireless protocols including EDGE, EV-DO, HSPA, and oth
`CS.
`System storage 116 may include an area for storing appli
`cations, operating system portions, and data. It is contem
`plated that system storage 116 may be on a removable SD
`(secure digital) storage or other similar device and that the SD
`storage may include security features for holding critical
`pieces of information Such as credit card numbers and other
`similar information. Alternatively, system storage 116 may
`include conventional magnetic tapes or disks, optical disks
`such as CD-ROM, DVD, magneto optical (MO) storage or
`any other type of non-volatile storage devices suitable for
`storing large quantities of data. These latter storage device
`types may be accessed locally through a direct connection or
`remotely in the “cloud' through broadband interface 112 or
`data/voice interface 114 type network connections.
`While examples and implementations have been
`described, they should not serve to limit any aspect of the
`exemplary embodiments. Accordingly, implementations of
`the exemplary embodiments can be implemented in digital
`electronic circuitry, or in computer hardware, firmware, Soft
`ware, or in combinations of them. Apparatus can be imple
`mented in a computer program product tangibly embodied in
`a machine readable storage device for execution by a pro
`grammable processor, and method steps of the invention can
`be performed by a programmable processor executing a pro
`gram of instructions to perform functions of the invention by
`operating on input data and generating output. The invention
`can be implemented advantageously in one or more computer
`programs that are executable on a programmable system
`including at least one programmable processor coupled to
`receive data and instructions from, and to transmit data and
`instructions to, a data storage system, at least one input
`device, and at least one output device. Each computer pro
`gram can be implemented in a high level procedural or object
`oriented programming language, or in assembly or machine
`language if desired; and in any case, the language can be a
`compiled or interpreted language. Suitable processors
`include, by way of example, both general and special purpose
`microprocessors. Generally, a processor will receive instruc
`tions and data from a read only memory and/or a random
`access memory. Generally, a computer will include one or
`more mass storage devices for storing data files; such devices
`include magnetic disks, such as internal hard disks and
`removable disks; magneto optical disks; and optical disks.
`Storage devices suitable for tangibly embodying computer
`program instructions and data include all forms of non-vola
`tile memory, including by way of example semiconductor
`memory devices, such as EPROM, EEPROM, and flash
`memory devices; magnetic disks Such as internal hard disks
`and removable disks; magneto optical disks; and CD ROM
`disks. Any of the foregoing can be supplemented by, or incor
`porated in, ASICs.
`When transmitting modulated acoustic carrier signal 111,
`the receive device 103 may encounter noise and signal deg
`radation as measured from the receive device microphone
`109, as shown in FIGS. 2A-2B.
`FIG. 2A is a graph illustrating a typical noise spectrum as
`measured by the receive device microphone. FIG. 2B is a
`graph illustrating levels of the modulated acoustic carrier
`signal 111 wirelessly received over the air at various frequen
`
`Page 21 of 28
`
`

`

`7
`cies versus noise levels. The X-axis in both Figures represents
`frequencies from 0 to 22,000 hertz, and the y-axis represents
`noise level in decibels relative to a full scale digital signal
`(dBFS). As shown in FIG. 2B, in one aspect of AMP 107, the
`modulated acoustic carrier signal 111 is transmitted at an
`inaudible acoustic frequency above 20,000 Hz, bringing
`modulated acoustic carrier signal 111 significantly above
`noise levels at those frequencies.
`Referring again to FIG.1, noise may originate from several
`Sources in the acoustics system 100 including the Sound com
`10
`ponents 104 (e.g., digital-to-analog converter, analog electri
`cal circuit, and speaker transducer), and a mechanical hous
`ing of the transmit device 101, the air medium; and Sources on
`the receive device 103 including mechanical housing and
`Sound components such as the microphone transducer, ana
`log-to-digital converter, and associated electrical circuitry.
`Each of these segments of the acoustic system will transform
`the modulated acoustic carrier signal 111 in Some way. These
`transformations could be considered a degradation of the
`intended signal. Possible artifacts may include: amplification
`or attenuation, alteration of the frequency response, adding
`noise (thereby reducing SNR), adding distortion, or altering
`the phase.
`FIG. 2C is a graph depicting an end-to-end system
`response, where the X-axis represents frequencies from 0 to
`22,000 hertz, and the y-axis represents the system response in
`dBFS (-15 to -90). In accordance with an exemplary embodi
`ment, the acoustic modulation protocol 107 enables a decode
`process on receive device 103 to compensate for an overall
`transfer function shown in FIG. 2C. In accordance with an
`exemplary embodiment, the acoustic modulation protocol
`107 may use a combination of filtering, dynamic gain, fre
`quency