`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`Address: COMMISSIONER FOR PATENTS
`PO. Box 1450
`Alexandria, Virginia ZZJIJ-MSO
`www.uspto.gov
`
`APPLICATION NO.
`
`FILING DATE
`
`FIRST NAMED INVENTOR
`
`ATTORNEY DOCKET NO.
`
`CONFIRMATION NOT
`
`15/269,776
`
`09/19/2016
`
`Alexander Kurganov
`
`10115-05709 US
`
`2723
`
`Patent Law Works, LLP
`310 East 4500 South, Suite 400
`Salt Lake City, UT 84107
`
`CHAWAN,VIJAY B
`
`ART UNIT
`2658
`
`PAPER NUMBER
`
`NOTIFICATION DATE
`
`DELIVERY MODE
`
`I 0/ I 9/201 8
`
`ELECTRONIC
`
`Please find below and/or attached an Office communication concerning this application or proceeding.
`
`The time period for reply, if any, is set in the attached communication.
`
`Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
`following e-mail address(es):
`docketin g@patentlawworks.net
`patents@patentlawworks .net
`
`PTOL-90A (Rev. 04/07)
`
`Google Ex 1030 - Page 1
`
`Google Exhibit 1030
`
`Google V. Parus
`
`Google Ex 1030 - Page 1
`
`

`

`
`
`United States Patent and Trademark Office
`
`Oflice ofthe ChiefFinancial Ofiicer
`
`October 10, 2018
`
`PATENT LAW WORKS LLP
`Attn: REENA KUYPER
`310 EAST 4500 SOUTH
`SUITE 400
`
`SALT LAKE CITY, UT 84107
`
`Dear Sir/Madam,
`
`The United States Patent and Trademark Office (USPTO) has reviewed your refund request for Reference
`Number 15269776 - Refund Request ID REFND-20181009-00012. Below is the current status of your
`refund request assigned to the following processing area: Office of Patent Examination Support Services
`
`
`
`
`-m Decision Reason(s)
`
`1806
`Approved
`Entity Status Change
`
`
`1806
`
`Approved
`
`Entity Status Change
`
`Refund Amount
`$120.00
`
`$90.00
`
`
`
`
`
`
`
`Generally, approved refunds will be issued in the form that the original payment was provided. You
`will receive a separate decision letter for any additional fee codes assigned to other processing areas, if
`applicable. Decisions may be subject to change if an error is identified.
`
`For questions related to this decision, contact the Case Resolution Team at (571) 272-1720 or
`case.resolution@USPTO.GOV.
`
`Thank you,
`Refund Branch
`
`PO. Box 1450, Alexandria. VA 22313-1450 - www.uspto.gov
`
`Google Ex 1030 - Page 2
`
`Google Ex 1030 - Page 2
`
`

`

`Document code: WFEE
`
`United States Patent and Trademark Office
`Sales Receipt for Accounting Date: 10/10/2018
`
`DMARTINO
`
`RF #30220114 Mailroom Dt: 10/10/2018
`
`15269776
`
`Credit Card Refund Total:
`
`$120.00
`
`American Express
`
`XXXXXXXXXXXX1 002
`
`Google Ex 1030 - Page 3
`
`Google Ex 1030 - Page 3
`
`

`

`Document code: WFEE
`
`United States Patent and Trademark Office
`Sales Receipt for Accounting Date: 10/10/2018
`
`DMARTINO
`
`RF #30220113 Mailroom Dt: 10/10/2018
`
`15269776
`
`Credit Card Refund Total:
`
`$90.00
`
`American Express
`
`XXXXXXXXXXXX1 002
`
`Google Ex 1030 - Page 4
`
`Google Ex 1030 - Page 4
`
`

`

`UNITED STATES PATENT AND TRADEMARK OEEICE
`
`
`
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`Address: COMIVIISS IONER FOR PATENTS
`P.O Box 1450
`Alexandria, Virginia 22313-1450
`www.uspt0 gov
`
`APPLICATION NO.
`
`15/269776
`
`ISSUE DATE
`
`10/09/2018
`
`PATENT NO.
`
`100963 20
`
`ATTORNEY DOCKET NO.
`
`CONFIRMATION NO.
`
`101 15-05709 US
`
`93219
`
`7590
`
`09/19/2018
`
`Patent Law Works, LLP
`310 East 4500 South, Suite 400
`Salt Lake City, UT 84107
`
`The projected patent number and issue date are specified above.
`
`ISSUE NOTIFICATION
`
`Determination of Patent Term Adjustment under 35 U.S.C. 154 (b)
`(application filed on or after May 29, 2000)
`
`The Patent Term Adjustment is 0 day(s). Any patent to issue from the above-identified application will include
`an indication of the adjustment on the front page.
`
`If a Continued Prosecution Application (CPA) was filed in the above-identified application, the filing date that
`determines Patent Term Adjustment is the filing date of the most recent CPA.
`
`Applicant will be able to obtain more detailed information by accessing the Patent Application Information
`Retrieval (PAIR) WEB site (http://pair.uspto. gov).
`
`Any questions regarding the Patent Term Extension or Adjustment determination should be directed to the
`Office of Patent Legal Administration at (571)-272-7702. Questions relating to issue and publication fee
`payments should be directed to the Application Assistance Unit (AAU) of the Office of Data Management
`(ODM) at (571)-272-4200.
`
`APPLICANT(S) (Please see PAIR WEB site http://pairuspto gov for additional applicants):
`
`Alexander Kurganov. Buffalo Grove, IL;
`Parus Holdings, Inc., Bannockburn, IL;
`
`The United States represents the largest, most dynamic marketplace in the world and is an unparalleled location
`for business investment, innovation, and commercialization of new technologies. The USA offers tremendous
`resources and advantages for those who invest and manufacture goods here. Through SelectUSA, our nation
`works to encourage and facilitate business investment. To learn more about why the USA is the best country in
`the world to develop technology, manufacture products, and grow your business, visit SelectUSA.gov.
`
`IR103 (ReV. 10/09)
`
`Google Ex 1030 - Page 5
`
`Google Ex 1030 - Page 5
`
`

`

`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`Address: COMMISSIONER FOR PATENTS
`PIO. Box 1450
`AlcxnndriII‘ Virginia 22313-1450
`www,usptoIgov
`
`APPLICATION NO.
`
`FILING DATE
`
`FIRST NAMED INVENTOR
`
`ATTORNEY DOCKET NO
`
`CONFIRMATION NO.
`
`15/269,776
`
`09/19/2016
`
`Alexander Kurganov
`
`10] 15—05709 US
`
`2723
`
`7590
`
`09l|8l20|8
`
`Patent Law Works, LLP
`310 East 4500 South, Suite 400
`Salt Lake City, UT 84107
`
`CHAWAN, VIJAY B
`
`ART UNIT
`2658
`NOTIFICATION DATE
`
`09/ I 8/201 8
`
`PAPER NUMBER
`
`DELIVERY MODE
`
`ELECTRONIC
`
`Letter Withdrawing a Notice Requiring Inventor’s Oath or Declaration
`
`The Notice Requiring Inventor’s Oath or Declaration mailed on 5 ’39-” was sent in error, and is
`hereby withdrawn. The time period set forth in the Notice of Allowance and Fee(s) Due to file a reply and
`pay the required fees continues to run from the mailing date of the Notice of Allowance and Fee(s) Due.
`Any time period set forth in the Notice of Allowability continues to run from the mailing date of the
`Notice of Allowability.
`
`Questions relating to this Notice should be directed to the Application Assistance Unit at 571—272-4200.
`
`(571)-272-4200 or 1(888)-786-0101
`
`Patent Publication Branch
`
`Office of Data Management
`
`Google Ex 1030 - Page 6
`
`Google Ex 1030 - Page 6
`
`

`

`‘ 101111
`“IONS 1‘1“
`
`
`
`'respendence Adams at 1111cattcd 1111
`In:ther coi-espnudcnu. 11111111”-
`nee flees 11/111131. mtniec'. to the cum;
`. r:
`
`
`
`
`
`
`
`
`below or alir
`".11 111111119
`1 ADDRF ‘5” for m intenance fee notit
`; and/or (11) indicating a separate "IT
`81001; 1 by :51) xpcnityne a new C0111: sponucnccadd
`,
`4 for domestic ma 111315 011111:
`
`' iiock 1 for any 91171113: 11fadd1'ess}
` liN'I CORRESPONDENCE ADDRESS (Nora ‘
`131 be used for any other accompanying
`J 11111151111111.7111“ cerut}
`
`
`
`
`£1011 additmnal paper
`.
`‘gnment 0r forma] drawing. must
`have 1 s own cemncate of 111.11
`
`
`1111?, or Liansmissmn
`
`932.19
`
`"”90
`y
`' 1
`.
`f
`3
`E ‘ant {A “I V» 0113, 1-1-41-
` 3.10 E .:-' 4500 South
` .116 413-0
`Salt Lake City. UTAH 84107
`UNI .131} STATES OF AMERICA
`
`{WM/".0112
`/
`
`PAR’I' B - FEES) 'T‘RANSMI’I'TAL
`
`Complete and send this farm, tegether with zappiimble [66(8), to: Mail MaiI Step ISSUE FEE
`Cummissiener fer Patents
`IMO. Box 145111
`Alexandria, Virginia 22313-1451}
`
` INSTRI.
`
`Ceitit‘iwto of Maiiing 131" "Imusmission
`111:2,:81?)! Cmtity that this F
`'
`1:. (19130511011 with t:
`
`States Postal Service with
` ' .d 1c 1 . 1.11310 (57117.73 7.8
`
`
`class 111.111111 2111
`
`FILING DATE
`
`
`
`
`
`FIRST NAMED INVENTOR
`ATTO 1\E"DOCKET N0.
`APPIICATI JN N.0
`
`
`
`
`
`
`
`Alexander Kurganov
`10115-05709 US
`09/19/2010
`15/269. 1
`I (1
`
`'I'I'I‘III 0.17 INVIZ.‘ T‘ION: ROIKIIS'I' VOICE? BRIYWSISR SYS'I'IZM AND VINCE ACTIVA'I‘ISI) DEVICE CONTRINJER
`
`
`ENTITY STATUS
`ISSUE FEE DUE
`PUBLICATION FEE DUE
`PREV. PAID ISSUE FEE
`
`
`
`11011;:10visionai
`UNDISCOUNTED
`$1000
`$0.00
`$0.00
`$1000
`081112018
`
`EXAM 19115.1(
`CLASSS 113C SASS
`
`CHAWAN, \"IJAY B
`2,158
`7041-2750011
`
`
`pondence address or 111d:
`01 'Fee Address" (37
`For p nting en the patent front page,
`gum
`
`
` . 31.1111115:y 5
`(111 12.63:
`
`(I) The names of up t1:- 3 :‘egtste .
`IV
`or agents 0R alternatively.
`
`
`
`an“ address 101‘ (hange of Cort‘espon
`J (flange of con‘espun a
`Addtmx!\t‘m PIG/ST’1
`attached.
`ed patent attmneys 01‘ agents. Ii 1101mm: is
` 111011 (or "Fee Address" Indication 101111 PTO/
`
`
`JFee Add1ess" 111111
`d. 11-.) n .1111e V1111 be 1:1intec.
`
`chad. Use of 11 Customer
`
`S8147; Rev 3-02 or 1 He recent) 21 2
`
`Number1” manned.
`
`
`[18311;\I‘E IxAVIF. AND RESIDENCE DATA TO BE PRINTED ON THE FATE T'[1111 t or type)
` 111111434] below. the document has been filed [or recordntion
`
`
`PLEASEIxOTE: U111::ts an assign e is ident 1rd 13-1:an no as\
`‘."data will appear on the patent 11 an :tssignee is
`as,set [111111in 37 CFR’J 11. Completion of this formis NOT a substitute for iilin" an assignment.
`
`(A) NAME OE ASSHJNEE.
`(B) RESIDE ICE- (LITY and STATE OR. COUNTRY)
`
` "Ance
`
`
`
`Paws. Hoidings, 111G.
`I
`V
`n c
`4
` signe1: I'm ppii‘q‘vri at re 71y or 171mm 'a
`J 11111
`'
`
`
`
`
`MI!“ 1.0. 1‘ pt
`":13
`11011 math: pr" "
`nt
`
`
`
`
`
`
`ratify JCOV'
`..
`
`
`
`
`
`
`
` ,Payment of F at (Please first ream)” any previously paid i Lie fee shown above) 4a. The fe‘flowing f;
`
`
`& Issue Fee
`U A check 1:: enclc-sed.
`
`Bannockbum. lilimie
`
`E1 l‘ub‘iication Fee (No smal‘i entity discount permitted)
`:3 Advance Order — # of C1.
`
`
`3.(.Imnge'111 If.“titty Status (110.11 status indie316d abovet
`
`E Pay111e11t by credit card. QQNXKQSNXMQQ
`(1101 1:: 1‘1
`
`yment, to
`
`
`
`
`
`
`w} Applitant 1ert1f§ing 1111110 entity status See 37 [FR 1.29
`
`See '37 CPR 1.27
`
`:3 Applicant assert 111g 5111311911111)? st
`
`J Applicant chain-Jging to regular Ltndiscottnted lee status.
`
`
`
`the risk of application abandonn“
`
`.tus, Che "11mg this box W111 be taken
`1‘0 entity 3
`
`
`
`
`
`t 10 {mere
`11y status.
`e taken tr: be a notificatmn of loss of entitlement to 3111311 (11‘ micro
`
`
`
`"IF
`
`
`
`
`
`
`An 0101.
` /Ree11a Kuyper/
`
`
` Typed c: p '
`name Reena Kuvper
`
`PTOI- 85 Par- 8 (10-13,) Approved for use tl‘n‘ough 10/31/2013.
`
`011311141031
`
`
`ILS. Patent and Trademark 0111:1215. D‘EPARTI‘M "1‘ (1‘12 COMPvtf:Rt1:
`
`Google Ex 1030 - Page 7
`
`Google Ex 1030 - Page 7
`
`

`

`Electronic Patent Application Fee Transmittal
`
`T't'e °f "New”:
`
`ACQUIRING INFORMATION FROM SOURCES RESPONSIVE TO NATURALLY—
`SPOKEN—SPEECH COMMANDS PROVIDED BY A VOICE—ENABLED DEVICE
`
`First Named Inventor/Applicant Name:
`
`Alexander Kurganov
`
`Attorney Docket Number:
`
`10115—05709 US
`
`Filed as Small Entity
`
`Filing Fees for Utility under 35 USC 111(a)
`
`Sub-Total in
`Description USD($) Quantity
`
`
`
`
`
`
`
`Basic Filing:
`
`Miscellaneous-Filing:
`
`Petition:
`
`Patent-Appeals-and-lnterference:
`
`Post-Allowance-and-Post-lssuance:
`
`
`UTILITY APPL ISSUE FEE
`
`Google Ex 1030 - Page 8
`
`Google Ex 1030 - Page 8
`
`

`

`Extension-of—Time:
`
`Total in USD ($)
`
`Miscellaneous:
`
`Google Ex 1030 - Page 9
`
`Google Ex 1030 - Page 9
`
`

`

`Electronic Acknowledgement Receipt
`
`“—
`
`——
`
`T'tle °f '"Venmm
`
`ACQUIRING INFORMATION FROM SOURCES RESPONSIVE TO NATURALLY—
`SPOKENESPEECH COMMANDS PROVIDED BY A VOICEEENABLED DEVICE
`
`
`
`“—
`
`Utility under 35 USC 111(a)
`Application Type:
`
`
`Payment information:
`
`The Director ofthe USPTO is hereby authorized to charge indicated fees and credit any overpayment as follows:
`
`——
`
`Google Ex 1030 - Page 10
`
`Google Ex 1030 - Page 10
`
`

`

`
`
`Document
`Number
`
`Information:
`
`the application.
`
`New Applications Under 35 U.S.C. 111
`lfa new application is being filed and the application includes the necessary components for a filing date (see 37 CFR
`1.53(b)—(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this
`Acknowledgement Receipt will establish the filing date of the application.
`National Stage of an International Application under 35 U.S.C. 371
`If a timely submission to enter the national stage of an international application is compliant with the conditions of 35
`U.S.C. 371 and other applicable requirements a Form PCT/DOIEOI903 indicating acceptance of the application as a
`national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.
`New International Application Filed with the USPTO as a Receiving Office
`lfa new international application is being filed and the international application includes the necessary components for
`an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number
`and of the International Filing Date (Form PCT/ROI105) will be issued in due course, subject to prescriptions concerning
`national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of
`
`Document Descri
`
`tion
`
`p
`
`File Size(Bytes)I
`Message Digest
`178863
`
`Pages
`Multi
`Part I.zip (if appl.)
`
`Issue Fee Payment (PTO—85 B)
`
`05709US_20'| 80829_|SsueFeeT
`ransmittal pdf
`
`6b573132ef4eb2627329a93bd3l939f4009
`(16:1
`
`Fee Worksheet ($806)
`
`feeainfopdf
`
`63159222137451c1951b1931544mf5bl160
`deOO
`
`
`
`Information:
`
`Total Files Size (in bytes)
`
`209304
`
`This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,
`characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a
`Post Card, as described in MPEP 503.
`
`Google Ex 1030 - Page 11
`
`Google Ex 1030 - Page 11
`
`

`

`IN THE UNITED STATES
`PATENT AND TRADEMARK OFFICE
`
`FIRST INVENTOR:
`
`Alexander Kurganov
`
`APPLICATION NO:
`
`15/269,776
`
`FILING DATE:
`
`September 19, 2016
`
`TITLE:
`
`Acquiring Information from Sources Responsive to Naturally-
`
`Spoken-Speech Commands Provided by a Voice-Enabled Device
`
`EXAMINER:
`
`Vijay B. Chawan
`
`GROUP ART UNIT:
`
`2658
`
`ATTY. DKT. NO:
`
`10115-05709 US
`
`CONFIRMATION NO:
`
`2723
`
`CERTIFICATE OF EFS-WEB TRANSMISSION UNDER 37 CFR 1.8
`[hereby certify that this con'espondence is being transmitted via the US. Patent and Trademark Office electronic filing
`system (EFS-VVeb) to the USPTO 011 8/29/2018
`
`Reena Kuyper, Reg. No. 33,830
`
`By:
`
`/Reena Kuyper/
`
`COMMISSIONER FOR PATENTS
`PO. BOX 1450
`
`ALEXANDRIA, VA 22313-1450
`
`INFORMATION DISCLOSURE STATEMENT UNDER 37 C.F.R 1.56 d
`
`Sir:
`
`Pursuant to 37 CFR § 1.56 ((1), Applicant makes a record in this prosecution history of
`
`references cited and considered by Examiners in related patent applications as designated below.
`
`Copies of these references are available in the respective file histories noted below.
`
`Respectfully, the filing of this information disclosure statement should not be construed as an
`
`admission that the information cited in this statement is, or is considered to be, material to
`
`patentability (under 37 C.F.R. § 1.56 (h)) of the allowed claims here as defined by 37 C.F.R. §
`
`Application No. 15/269,776
`
`Dkt. No. 10115-05709 US
`
`Page 1 of 3
`
`Google Ex 1030 - Page 12
`
`Google Ex 1030 - Page 12
`
`

`

`1.56 (b).
`
`In addition, certain references are papers from prior inactive litigation proceedings of
`
`related patents.
`
`Specifically, the following listed references were cited in parent U.S. Patent Application
`
`Number 13/462,819, from which the present application claims priority, said U.S. Patent
`
`Application issued as U.S. Patent No. 9,451,084 B2:
`
`Patent Number
`
`Issue Date
`
`Patentee
`
`5145452
`
`1992—09—08
`
`Chevalier
`
`2001-03- 13
`
`2002-12-31
`
`Greenspan
`
`Bareis et al.
`
`
`
`
`
`Foreign Document
`Number
`97/3748 1
`WO
`1997-10-09
`Northern Telecom
`
`Limited
`
`Country Code
`
`
`
`Publication Date
`
`Name of Applicant
`
`98/2305 8
`
`WO
`
`1998—09—03
`
`Premiere
`
`Communications, Inc.
`
`Non-Patent Literature Documents
`
`
`HEMPHILL et al., “Surfing the Web by Voice,” ACM Multimedia 95 — Electronic
`Proceedings, November 5-9, 1995, San Francisco, CA, 8 pgs.
`OYE, “Juggler,” available at http://www.philoye.comiwork/juggler2.shtml (accessed on
`December 8, 2006), 1
`.
`
`OYE, “Juggler,” available at http://www.philoye.eomlwork/juggler3.shtml (accessed on
`December 8, 2006), 1 pg.
`Plaintiff Parus Holdings, lnc.’s Supplemental Response to Defendant Web Telephony LLC’s
`Second Set of Interrogatories (Nos. 13-17), Parus Holdings, Inc. v. Web Telephony LLC &
`Robert Swartz, Case No. 06-cv-01146 (ND. 111.), October 31, 2006, 31 pgs.
`
`Application No. 15/269,776
`
`Dkt. No. 10115-05709 US
`
`Page 2 of 3
`
`Google Ex 1030 - Page 13
`
`
`
`6201814
`
`6501966
`
`6618039
`
`6633 846
`
`2003—09—09
`
`Grant et al.
`
`2003- 10- 14
`
`Bennett
`
`
`
`Google Ex 1030 - Page 13
`
`

`

`The following listed reference was cited in US. Application Number 15/436,377, which
`
`is a continuation of the present application:
`
`
`
`Non-Patent Literature Documents
`
`Judgment without Opinion for Parus Holdings Inc., V. Sallie Mae Bank, Navient Solutions
`
`Inc., PNC Bank, N.A., Suntrust Bank, Suntrust Mortgage Inc., 2016—1179, 2016—1180, 2016— 1181, entered February 27, 2017, 2 pgs.
`
`For convenience, copies of the foreign and N PL references are provided.
`
`This submission is made with payment of the fee set forth in 37 C.F.R. § 1.17 (p). The
`
`Examiner is respectfully requested to include this information disclosure statement as part of the
`
`prosecution history of the present application.
`
`Respectfully submitted,
`PARUS HOLDINGS, INC.
`
`Dated:
`
`August 29, 2018
`
`By:
`
`RBK/kdl
`
`/Reena Kuyper/
`Reena Kuyper, Reg. No. 33,830
`Of
`PATENT LAW WORKS LLP
`
`310 East 4500 South, Suite 400
`
`Salt Lake City, UT 84107
`Tel.: (650) 537—4509
`Fax: (801) 355-0160
`Email: flaw Eerie? vatentl a wworlsxs .uet
`
`Application No. 15/269,776
`
`Dkt. No. 10115-05709 US
`
`Page 3 of 3
`
`Google Ex 1030 - Page 14
`
`Google Ex 1030 - Page 14
`
`

`

`EP 02010210 @
`
`PCT
`
`lntemational Bureau
`WORLD IN'lELu-zcrUAL PROPERTY ORGANIZATION
`
`
`
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`51) International Patent Classification 6 =
`(11) International Publication Number:
`WO 97/37481
`
`H04M 1/27, GlOL 5/06
`
`(43) International Publication Date:
`
`9 October 1997 (09.10.97)
`
`(21) International Application Number:
`
`PCT/CA97/00008
`
`(8i) Designated States: CA. European patent (AT. BE. CH, DE,
`DK, ES, Fl. FR, GB. GR, lE. lT. LU. MC, NL, PT, SE).
`
`(22) International Filing Date:
`
`9 January 1997 (09.01.97)
`
`Published
`
`(30) Priority Data:
`08/623,635
`
`28 March l996 (28.03.96)
`
`US
`
`With international search report.
`
`(71) Applicant: NORTHERN TELECOM LIMITED [CA/CA];
`World Trade Center of Montreal, 8th floor, 330 St. Antoine
`Street West. Montreal, Quebec HZY 3Y4 (CA).
`
`02) Inventor: WONG. cm; Apartment #3390. 2850 Middlefield
`Road. Palo Alto. CA 94306 (US).
`
`(74) Agent: MacGREGOR. George; Northern Telecom Limited.
`Patent Dept. PO. Box 351i, Station "C". Ottawa. Ontario
`KIY 4H7 (CA).
`
`
`
`(54) Title: APPARATUS AND METHOD FOR REDUCING SPEECH RECOGNITION VOCABULARY PRPLEXITY AND DYNAM-
`ICAILY SELECTING ACOUSTIC MODELS
`
`(57) Abstract
`
`A method of reducing the perplexity of a speech recognition vocabulary and dynamically selecting speech recognition acoustic model
`sets used in a simulated telephone operator apparatus. The directory of users of the telephone network is subdivided into subsets wherein
`each subset contains the names of users within a certain location or exchange. A speech recognition vocabulary database is compiled for
`each subset and the appropriate database is loaded into the speech recognition apparatus in response to a requested call to the location
`covered by the subset. Furthermore, a site-specific acoustic model set is dynamically loaded according to the location of a calling party.
`An apparatus for carrying out the method is also discussed.
`
`Google Ex 1030 - Page 15
`
`Google Ex 1030 - Page 15
`
`

`

`
`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the from pages of pamphlets publishing
`tundra
`LS
`5min
`LT
`Lithuania
`Finland
`LU
`manna;
`France
`Latvia
`LV
`Gabon
`Monaco
`MC
`United Kingdom
`MD
`Republic of Moldova
`Georgia
`MG
`Madagaacar
`Ghana
`MK
`The former Yugoslav
`Guinea
`Republic of Macedonia
`Mali
`Mongolia
`Mmr'unia
`Malawi
`Mexico
`Mm
`Netherlmds
`Norway
`New Zealand
`Poland
`Pormgal
`Rommia
`Russian Federation
`Sudan
`Sweden
`Singapore
`
`Albania
`Armenia
`Austria
`Auwflia
`Ambaijan
`30min and Herzegovina
`BM
`
`Congo
`Switzerland
`Cue d‘lvoire
`
`:::::::§:
`93328
`
`BfiEQSQQQQBQ
`
`"nus-I7
`
`
`
`SEESEEfifififi
`
`Democratic People's
`Republic 0! Korea
`Republic of Korea
`ml!
`Sal‘nl Lucia
`Liechtmtein
`Sl'i Lilla
`Liberia
`
`ML
`
`aaegsaaasaaagga
`
`Google Ex 1030 - Page 16
`
`inlemalional applications under the PCT.
`
`Slovenia
`Slovakia
`Senegal
`Swaziland
`Chad
`T030
`Tajikistan
`Turkmenistan
`May
`1‘r‘nidad and Tobago
`Ukraine
`Uganda
`Un'ned Swen cl America
`Uzhdiaun
`Viet Nam
`Yugoslavia
`Zimbabwe
`
`gassgguaz=aaaaaw
`
`2
`
`Google Ex 1030 - Page 16
`
`

`

`WO 97/37481
`
`PCT/CA97/00008
`
`APPARATUS AND METHOD FOR REDUCING SPEECH RECOGNITION
`VOCABULARY PERPLEXITY AND DYNAMICALLY SELECTING
`ACOUSTIC MODELS
`
`. J:
`
`E
`
`I
`
`.
`
`This invention relates to automatic speech
`
`recognition in telecommunication systems and to the use of
`
`such systems to provide large scale voice activated dialing
`
`10
`
`and information retrieval services.
`
`W I
`
`n the early development of telephone systems it
`
`was commonplace for a telephone subscriber to converse
`
`15
`
`directly with a telephone operator at a telephone central
`
`office. The telephone subscriber would verbally request the
`
`telephone operator to complete a connection to a called
`
`party. As telephone exchanges were small
`
`the telephone
`
`operator was aware of virtually all of the subscribers by
`
`20
`
`name and manually completed the requested connection. with
`
`the advent of dial telephone services, calls within an
`
`exchange could be completed automatically, and only certain
`
`toll calls required operator assistance.
`
`Today, operator
`
`assisted calls have become the exception and are usually
`
`25
`
`comparatively expensive. Machine-simulated operator
`
`functions,
`
`including limited speech recognition services,
`
`have recently been available for expediting some typical
`
`operator—assisted functions. This includes “collect“ long
`
`distance calls wherein completion of the connection is
`
`30
`
`contingent upon the called party agreeing to pay for the
`
`service. However,
`
`these functions are limited to the simple
`
`recognition of "yes" or “no" so there is little room for
`
`non-functionality due to uncertainty as to which word was
`
`spoken. There have also been advancements in voice
`
`35
`
`recognition services relating to directory assistance_but
`
`these too are directed to a very limited vocabulary.
`
`Google Ex 1030 - Page 17
`
`Google Ex 1030 - Page 17
`
`

`

`W0 97/3748]
`
`PCT/CA97I00008
`
`12mm
`
`The prior art contains several recent developments
`pertaining to voice recognition in general, and to voice
`recognition applicable to telecommunication systems in
`
`particular.
`
`U.S. Patent No. 5,091,947, which issued February
`
`25, 1992 to Ariyoshi et a1, entitled "Speech Recognition
`Method and Apparatus", discloses a voice recognition system
`for comparing both speaker dependent and speaker independent
`utterances against stored voice patterns within a
`
`coefficient memory.
`The voice identification comparator
`selects the one voice pattern having the highest degree of
`
`similarity with the utterance in question.
`
`10
`
`15
`
`In U.S. Patent No. 5,165,095, which issued on
`
`November 17, 1992, Borcherding discloses a voice recognition
`system to initiate dialog to determine the correct telephone
`number. According to the '095 patent,
`the calling party is
`
`20
`
`25
`
`first identified so that a database containing speaker
`templates can be accessed. These templates are then used to
`compare the dial command so that the dialing instructions
`can be recognized and executed.
`An example of a dialing
`directive in the patent is "call home", with "call" being
`the dial command and "home“ being the destination
`
`identifier.
`
`30
`
`35
`
`in U.S. Patent No. 5,390,278 issued
`Gupta et al,
`February 14, 1995, disclose a flexible vocabulary speech
`recognition for recognizing speech transmitted via the
`public switched telephone network. This voice recognition
`technique is a phoneme based system wherein the phonemes are
`
`modeled as hidden Markov models.
`
`the
`In spite of these ongoing developments.
`functionality of automatic recognition of human speech by
`machine has not advanced to a degree wherein a calling party
`
`Google Ex 1030 - Page 18
`
`Google Ex 1030 - Page 18
`
`

`

`W0 97/3748]
`
`PCT/CA97100008
`
`3
`
`can simply speak the called party's name and thereafter be
`
`connected as reliably as a human operator in situations
`
`where the database for a potential called party is very
`
`large (greater than 100 names).
`
`W T
`
`he present invention is in the field of human
`
`speech recognition performed by machines and more
`particularly relates to a reduction of the perplexity of the
`
`speech recognition task in the context of names spoken by a
`
`telephone user in a telephone system.
`
`Individual users of telephone networks are divided
`
`into subsets to facilitate identification of the vast number
`
`of subscribers to the service.
`
`In the public network these
`
`subsets are local exchanges. Private switching networks
`
`such as the Nortel Electronic Switching Network (ESN)
`
`assigns individual ESN numbers to each location within the
`
`private network.
`
`The present invention relies on these
`
`subsets or location identifiers to reduce the perplexity of
`
`a speech recognition application.
`
`Therefore in accordance with a first aspect of the
`
`present
`
`invention.
`
`there is provided a telephone network
`
`including a plurality of telephone exchanges, each for
`
`serving a plurality of telephone terminals and each being
`
`interconnected with at least one other of the.telephone
`
`exchanges for providing telephone communications between
`
`users of the telephone terminals. The network function
`
`includes a simulated telephone operator apparatus for
`
`receiving a speech request from a user for connection to
`
`another telephone user and to translate this request into a
`
`directory number for use by the appropriate one of the
`
`telephone exchanges.
`
`The translation is in accordance with
`
`a speech recognition algorithm and an active speech
`
`recognition vocabulary selected in accordance with the
`
`origin of the request.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`Google Ex 1030 - Page 19
`
`Google Ex 1030 - Page 19
`
`

`

`W0 97/3748]
`
`PCTICA97I00008
`
`4
`
`In an ESN application the active speech
`
`recognition vocabulary is limited to the names of the
`individuals serviced by the ESN number.
`In a preferred
`is
`embodiment the ESN number, which is also a location code.
`contained in the first two or three digits of the directory
`
`number.
`
`In accordance with a second aspect of the
`
`10
`
`15
`
`20
`
`invention there is provided a simulated telephone operator
`server for a telephone network.
`The server has means for
`storing voice utterances of a calling party telephone user
`and means responsive to location information in association
`with the telephone user for selecting an active speech
`recognition vocabulary.
`Speech detection means are provided
`for processing the stored voice utterance in accordance with
`a speech recognition algorithm and the active speech
`recognition vocabulary. Directory lookup means identify a
`directory listing of a called party corresponding to a
`result of'the processing by the speech detection means.
`server also includes means for transmitting the directory
`listing to a telephone exchange serving the called party.
`
`The
`
`25
`
`30
`
`35
`
`In accordance with a further aspect of the
`invention there is provided a telephone exchange comprising:
`a plurality of ports for serving a plurality of telephone
`users'
`telephone instruments via telephone lines;
`a trunk
`facility for connection to another telephone exchange;
`a
`switching network for connecting and disconnecting the
`telephone instruments;
`a controller means for causing a
`
`switching network with a solicitation signal, and
`subsequently for being responsive to a telephone number
`received in association with the newly OFF HOOK telephone
`instrument for completing a telephone call via the switching
`network. The exchange also includes an originating register
`means for storing voice band signals received from the newly
`OFF HOOK telephone instrument via the switching network.
`
`Google Ex 1030 - Page 20
`
`Google Ex 1030 - Page 20
`
`

`

`WO 97/37481
`
`PCTICA97100008
`
`5
`
`Means are provided for detecting digits represented by
`
`frequency signals. within the stored voice band signals,
`
`in
`
`accordance with a standard for key pad dialed telephone
`
`numbers and for transmitting detected digits to the call
`
`controller.
`
`A simulated telephone operator apparatus
`
`receives and translates voice band signals in accordance
`
`with a speech recognition algorithm and an active speech
`
`recognition vocabulary selected in accordance with the
`
`origin of the voice band signals into a directory number for
`
`use by the controller means.
`
`An interface facility is
`
`provided for transmitting the stored voice band signals via
`
`the switching network to the simulated telephone operator
`
`server apparatus in an event wherein the voice band signals
`
`did not
`
`include a key pad dialed digit.
`
`In accordance with yet a further aspect of the
`
`present
`
`invention there is provided a method of detecting a
`
`voiced speech request of a calling party for connection to
`
`another user of an automatic telephone exchange.
`
`The method
`
`comprises storing a plurality of speech recognition
`
`vocabularies in association with geographic location of
`
`users;
`
`receiving the voiced request and information as to
`
`the geographic location of the user having voiced the
`
`request
`
`from the automatic telephone exchange;
`
`selecting an
`
`active speech recognition vocabulary in accordance with the
`
`information as to the geographic location of the user and,
`
`in accordance with a speech recognition algorithm and the
`
`selected active speech recognition vocabulary,
`
`translating
`
`the received request into a directory number for use by the
`
`automatic telephone exchange in setting up a telephone
`
`connection between the calling telephone user and the other
`
`telephone user;
`
`.
`
`E
`
`r.
`
`.
`
`E
`
`1
`
`.
`
`The invention will now be described in greater
`
`detail with reference to the attached drawings wherein:
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`Google Ex 1030 - Page 21
`
`Google Ex 1030 - Page 21
`
`

`

`W0 97/3748!
`
`PCTICA97100008
`
`6
`
`FIGURE 1 is a block diagram illustrating trunk
`
`connections between private switch locations;
`FIGURE 2 is a block diagram of the system hardware
`
`architecture;
`
`FIGURE 3 is an overall system state diagram; and
`
`FIGURE 4 is a state diagram of the key word
`
`handler.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`'l
`
`D
`
`'
`
`i
`
`h
`
`In
`
`' n
`
`The following description relates to an
`enterprise~wide speech directory calling service within a
`company or corporation having a number of locations.
`Each
`location is assigned a unique electronic switching network
`(ESN)
`location code or ESN number.
`As shown in the block
`diagram of FIGURE 1,
`the on-site PBX 20 at each location is
`connected to each other location via trunk connectors 22.
`In this discussion the ESN comprises a three—digit code to
`identify the location.
`It is to be understood, however,
`that it is not essential to use all three digits to identify
`the location as it may be sufficient to use the first two
`
`for example.
`
`FIGURE 2 illustrates the hardware architecture in
`accordance with a preferred embodiment of the invention. As
`shown, PBX 20 is connected to trunk 22 and to a plurality of
`on site telephone sets as known in the art. The speech
`recognition system 30 of the invention is connected to the
`PBX 20 via T1 line 32 via T1 card 34 and via signal link 36
`and signal link card 38.
`Speech recognition system 30
`includes a speech recognition processor operating on a
`speech recognition algorithm, central processor and control
`units as well as memory cards for storing active speech
`
`recognition vocabulary data bases.
`
`35
`
`Although FIGURE 1 refers to a private switching
`network using BSNs,
`it is to be understood that the
`
`Google Ex 1030 - Page 22
`
`Google Ex 1030 - Page 22
`
`

`

`W0 97/3748]
`
`PCTICA97/00008
`
`7
`
`invention is not limited to such networks but can also be
`
`adapted to use in public switching systems.
`
`One objective metric used to measure the accuracy
`
`of a speech recognition system is the Word Error Rate (WER).
`
`The WER is defined as the total number of incorrectly
`
`recognized words made by a speech recognition system divided
`
`by the total number of words spoken by a user of the system.
`
`10
`
`“4H? =
`
`NumberofErrorsMadebyRecognize;-
`
`NumberofWordsSpokenbyUser
`
`15
`
`20
`
`The present invention makes use of information as
`
`to the calling party's location for automatically assisting
`
`in improving the WER of a speech recognition system on a
`
`spoken called party's name for the pu