a2) United States Patent
`US 9,218,530 B2
`(10) Patent No.:
`
` Daviset al. (45) Date of Patent: Dec. 22, 2015
`
`
`US009218530B2
`
`(54) SMARTPHONE-BASED METHODS AND
`SYSTEMS
`
`(56)
`
`(75)
`
`Inventors: Bruce L. Davis, Lake Oswego, OR
`.
`‘
`(US); Tony F. Rodriguez, Portland, OR
`(US)
`
`(73) Assignee: Digimare Corporation, Beaverton, OR
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1059 days.
`
`(21) Appl. No.: 13/207,841
`(22)
`Filed:
`Aug. 11,2011
`
`References Cited
`U.S. PATENT DOCUMENTS
`6.122.403 A
`9/2000 Rhoad
`122,
`oads
`6.182.218 Bl
`1/2001 Saito
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`wo
`
`5/2012
`W02012061760
`OTHER PUBLICATIONS
`PCT Written Opinion of the Int’! Searching Authority, PCT/US11/
`g
`P
`ty.
`§9412 (WO12061760), Apr. 13, 2012.
`(Continued)
`
`(65)
`
`Prior Publication Data
`US 2012/0116559 Al
`May 10, 2012
`
`Primary Examiner — Joseph Saunders, Jr.
`(74) Attorney, Agent, or Firm — Digimarc Corporation
`
`ABSTRACT
`
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 13/174,258,
`filed on Jun. 30, 2011, now Pat. No. 8,831,279.
`(60) Provisional application No. 61/410,217,filed on Nov.
`nos
`oe
`4, 2010, provisional application No. 61/449,529, filed
`(Continued)
`
`(51)
`
`Int. Cl.
`GO6F 17/00
`G06K 9/00
`
`(2006.01)
`(2006.01)
`(Continued)
`
`57
`67)
`Methods and arrangements involving portable devices, such
`as smartphones and tablet computers, are disclosed. One
`arrangement enables a creator of content to select software
`with which that creator’s content should be rendered—assur-
`ing continuity between artistic intention and delivery.
`Another arrangementutilizes the camera of a smartphoneto
`identify nearby subjects, and take actions based thereon. Oth-
`ers rely on nearfield chip (RFID)identification of objects, or
`on identification of audio streams(e.g., music, voice). Some
`ofthe detailed technologies concern improvementsto the user
`interfaces associated with such devices. Others involve use of
`these devices in connection with shopping, text entry, sign
`(52) U.S. Cl.
`language interpretation, and vision-based discovery. Still
`CPC veeeserneess G06K 9/0067] (2013.01); GO6F 3/002
`other improvements are architectural in nature, e.g., relating
`(2013.01), GO6F 3/0236 (2013.01); GO6F
`to evidence-based state machines, and blackboard systems.
`3/0237 (2013.01); GO6F 304886 (2013.01);
`Yet other technologies concern use oflinked data in portable
`HO4M 1/7253 (2013.01);
`devices—someofwhich exploit GPU capabilities. Still other
`(Continued)
`recologiesconcer computational photography. great
`(58) Field of Classification Search
`CPC vevceessssssssseeveeee G10H 2240/041; G10L 19/018—-Variety of other Teatures and arrangementsare also detailed.
`USPC iceccsessseescrsscnseecscenssenscnssenssseecneeeanees 700/94
`
`See application file for complete search history.
`
`33 Claims, 20 Drawing Sheets
`
`wn
`
`GEFEVG
`
`CAPPOGORS
`
`SW SS584,
`
`SY PF 285 AND
`.
`.
`:
`EWSBAEBCesPhdeGlesks
`gAY Baek GH SWC9958 GR
`aE Ke
`SO
`SISA
`wae
`MAKE
`aw 2BBAC OR EW BEFAS
`4sen|Mipresses
`
`-
`
`,
`
`RE3SS
`BESSEA
`
`ANT.
`DAN TIME
`
`AGE 20-25
`AGE 26-25
`
`—oronessar RGDEC TPDEA
`
`a
`BESCAL
`
`-
`
`Alas
`
`1
`
`APPLE 1010
`
`APPLE 1010
`
`1
`
`

`

`US 9,218,530 B2
`
`Page 2
`
`Related U.S. Application Data
`
`on Mar. 4, 2011, provisional application No. 61/467,
`862, filed on Mar. 25, 2011, provisional application
`No. 61/471,651, filed on Apr. 4, 2011, provisional
`application No. 61/479,323, filed on Apr. 26, 2011,
`provisional application No. 61/483,555, filed on May
`6, 2011, provisional application No. 61/485,888, filed
`on May 13, 2011, provisional application No. 61/501,
`602, filed on Jun. 27, 2011.
`
`(51)
`
`Int. Cl.
`
`(52)
`
`(56)
`
`GO06F 3/00
`G06F 3/023
`GO6F 3/0488
`HO4M 1/725
`GIOL 19/018
`US. Cl.
`
`(2006.01)
`(2006.01)
`(2013.01)
`(2006.01)
`(2013.01)
`
`CPC....... G10H 2240/04] (2013.01); GIOL 19/018
`(2013.01); HO4M 2250/04 (2013.01); HO4M
`225/52 (2013.01)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,590,996 Bl
`6,614,914 Bl
`6,947,571 Bl
`6,968,564 Bl
`6,990,453 B2
`7,003,731 Bl
`7,006,555 Bl
`7,020,304 B2
`7,185,201 B2
`
`7/2003 Reedet al.
`9/2003 Rhoadsetal.
`9/2005 Rhoadsetal.
`11/2005 Srinivasan
`1/2006 Wang
`2/2006 Rhoads
`2/2006 Srinivasan
`3/2006 Alattar etal.
`2/2007 Rhoads
`
`7,359,889
`7,370,190
`7,486,827
`7,516,074
`7,760,902
`200 1/0024568
`2002/0044659
`2005/0232411
`2007/0064 562
`2007/0124756
`2007/0174059
`2007/0185840
`2007/02507 16
`2007/0253 594
`2007/0274537
`2008/0243 806
`2008/0263046
`2008/0276265
`2008/0300011
`2008/03 17278
`2009/0142038
`2009/0204640
`2009/0247 184
`2010/0048242
`2010/0119208
`2010/0150434
`2010/0205628
`20 10/0228632
`2011/0161076
`2011/0212717
`2011/02743 10
`2012/0046071
`
`B2
`B2
`B2
`B2
`B2
`AL*®
`Al
`Al
`Al*
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al*
`Al*
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`Al
`
`4/2008 Wang
`5/2008 Calhoon
`2/2009 Kim
`4/2009 Bilobrov
`7/2010 Rhoads
`9/2001 Mori ccc ee ceccccecereeeenes 386/96
`4/2002 Ohta
`10/2005 Srinivasan
`3/2007 Hanetal. ou... 369/53.2
`5/2007 Baluja etal.
`7/2007 Rhoadset al.
`8/2007 Rhoads
`10/2007 Brunk et al.
`11/2007 Lunt
`11/2007 Srinivasan
`10/2008 Dalal
`10/2008 Kristenssonet al.
`11/2008 Topchy
`12/2008 Rhoads
`12/2008 Lefebvre
`6/2009 Nishikawa .......c.e 386/109
`8/2009 Christensenetal. ...... 707/104.1
`10/2009 Sennett etal.
`2/2010 Rhoadset al.
`5/2010 Davis et al.
`6/2010 Reed
`8/2010 Davis et al.
`9/2010 Rodriguez
`6/2011 Davis et al.
`9/2011 Rhoadset al.
`11/2011 Rhoads
`2/2012 Brandisetal.
`
`OTHER PUBLICATIONS
`
`PCTInt’l Preliminary Report on Patentability, PCT/US2011/059412
`(published. as WO2012/061760), May 7, 2013.
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 1 of 20
`
`US 9,218,530 B2
`
` MEMORY
`
` METWIORR
`
`CONNECTION
`
`
`
` REMOTE
`
`COMPUTER
`
`
`
`
`SYSTEMIS}
`
`FIG. 1
`
`FG. 2
`
`
`
`smoons
`
`SAVBBSBA
`
`SVY FFRES AND
`
`30
`
`
`CONTENT ID|CONTEXT! | CONTEXTS SOFTWARE
`
`oo
`sere|usa|
`
`
`
`SW SS458evePbdmikego
`
`
` BY SEAS 1 OR Sw 09038 OR
` g
`
`SAAN SRBAC OR SW SRPAP
`
`
`
`
`COHHAD 77 DEA
`
`
`
`COMED FIBEC
`
`
`
`S2F
`
`x
`
`oo
`adh
`
`
`
`B33Cad
`
`BS SCaR
`
`3
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 2 of 20
`
`US 9,218,530 B2
`
`‘a
`tatoocateeaeaeeeeaeeeaeeeey
`ssaetatoetttoate
`
`xif:%fi3\f‘:4sisisivigi‘isi
`
`eins.okOfOTeEt?
`
` “8$04GLUEBY}$1MON
`
`aNIEEEELLLLLLLLLLEEDLELIE
`
`an*
`
`EDL
`
`seiienneineenieineenieineeniineenicneernel
`
`Nt
`
`yran\,iaaenn\|(\‘i)|,NL
`
`
`
`sateentettoedtio>aiorineninetaiinesianbananaiannnaannnaiinnaniunnnunanananabaninnattitaweMiweSS%,
`
`$°DidDid€Sid
`
`4
`
`
`
`
`
`

`

`U.S. Patent
`
`US 9,218,530 B2
`
`Dec. 22, 2015
`
`Sheet 3 of 20
`
`
`
`as
`“
`#
`Seok
`3
`ay
`’
`.
`*
`2
`‘a
`?
`3
`RRRoeaesBRFReeeSSSS
`Ca vw ¥ eR Ge y
`¥
`i
`G
`
`
`p enoiiiinnnnninininiiinnninninininiiininninnnininiin
`
`
`5
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 4 of 20
`
`US 9,218,530 B2
`
`
`
`$
`
`3
`
`vf
`
`FS E
`F eeirrriyy
`eteee
`
`
`ea 2 Cy
`
`
`
`
`
` €
`
`beperk |
`
`6
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 5 of 20
`
`US 9,218,530 B2
`
`SOOFUDAT
`
`PREY
`
`peHHoyAy
`
`
`
`asus{GURY
`
`
`
`+BaersSECIDOU
`
`
`
`oyCULARARAL
`
`AURABER
`
`wet
`Shopeerrr,
`Serre
`roa
`
`ann.
`
`
`
`RRMAMOREEPMOAIAMALAIEEAAALANAAALSAAALSALIALLEILAAAAIELASAAAIAAABONS,
`peaquaTig “pivinsnisnininiinnisninininsinisnisniinininninninnnisnisninisininnin
`BAUONBRAS ¢¢¢¢2
`
`pags
`
`PREES
`
`
`
`HOREpoautshiged
`
`aEayy
`
`>BPIDGUA
`
`OTStd
`
`ARC
`
`BOSUES
`
`7
`
`
`
`
`

`

`Sheet 6 of 20
`
`US 9,218,530 B2
`
`U.S. Patent
`
`Dec. 22, 2015
`
`
`
`8
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 7 of 20
`
`US 9,218,530 B2
`
`
`
`*
`ak
`fiG. 13
`
`%
`;
`
`z
`
`3
`
`FIG. 14
`
`imageSensara DF LZ
`
`imageSizes
`
`Oxd80
`
`imageBithepth=8
`
`aes
`
`ene
`
`one
`ponte enn
`
`imageType=RGB
`oePNa
`é
`
` wwe
`
`
`Pixel=(,0}
`we % ae
`a
`4
`RedValue GreenValue
`=§5
`#32
`
`sans,
`BlueValue
`246
`
`BlueValue
`=16
`
`af
`
`=33
`
`FIG. 15
`
`9
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 8 of 20
`
`US 9,218,530 B2
`
`
`
`Applications:
`
`Eveust
`Contraier
`
`Engine
`
`inference
`
`
`
`
`
`
`:
`genta
`
`
`Remostion
`
`f Recogeitios:
`Agen?
`
`FR. 46
`
`ikgitnarc ftscaver Apolicathos |
`
`FIG. 17
`
`
`
`
`BES demerit
`acon} tren |
`
`wiiien Chase
`
`inference
`Engine
`
`Digtmarce Discover bipleseniation
`Ss
`
`Wht
`
`
`RA: Audio
`
`Fingerprint
`
`LE Ra: Barcoute
`FE
`
`Bay image
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 9 of 20
`
`US 9,218,530 B2
`
` FIG. 18
`
`FIG. 19
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`US 9,218,530 B2
`
`Sheet 10 of 20
`
`EONAMSEGNANANEGNAANEANEANEAEANAAt
`
`EREanneateneneantinteneunt
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 11 of 20
`
`US 9,218,530 B2
`
`FIG. 22
`
`ra
`
`é G
`
`REEN
`
`
`
`o
`
`BLUE
`
`“
`
`“ROYAL BLUE PIXEL
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 12 of 20
`
`US 9,218,530 B2
`
`
`A
`
`a
`
`FIG. 23
`
`SUBIECT
`
`ROB HasChid TED
`
`FIG, 24
`“Conntninnonlin nonninnonlinno ttn nonnin notnin nstin nSttn telnet“4¢
`3
`.
`
`:
`‘
`—_
`| BOBHasBrothercHUcK oF,
`nou
`we OF
`fe
`esas
`we oes
`+ STEVE HasChild LARRY g-
`BOB Hascete TES pauhasChild IM FRANK HasChild 808
`
`_. ALICE HasSister HELEN
`
`NICK HasSister SUE
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 13 of 20
`
`
`
`US 9,218,530 B2
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 14 of 20
`
`US 9,218,530 B2
`
`FIG. 26 ~ PREDICATE TEMPLATES
`
`seaeaiglonenlgevei
`x
`
`ORECTIb} Haslincte
`
`iectie} warE
`Ohjectid) | ORIECTg} Hewincde
`Ghjent{a}
`
`CHUECTIa} Hastinds
`iMalectic}
`
`CBFECTIOS Haslinele
`Obiectic}
`
`GWECT(ad} Hastincks
`
`i OBIECTej Newnee
`BMATCR SOR Objectic}
`
` -
`
`oe
`
`maneiein
`
`LE
`
` matens SL|| OBCH)Ha
`
`GBIECTc} Hastings
`Cibeet{s}
`
`| OOIECTin} Haslinels:
`. Em Object{al}
`
`ORIECTIN Haslinde
`Cibject{d}
`
`4 ORMECHEs} HanUacle
`
`Objectic}
`
`18, cavern) Mastince
`
`Ohyectic}
`
`QBJECTH) Haskincle
`Objectic}
`
`CHUECTia} Hastincde
`
`lomdend GBLECTIA) Hastinde
`MATCH SE Objectid)
`13]
`LL OBIETTYG} Haskineks
`MAH IST|| otieettds
`Seb
`
`
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 15 of 20
`
`US 9,218,530 B2
`
`FIG. 28
`
`
`
`bz,
`
`WEL 2 OPTONSLEY
`7G DISPLAY
`
`SEARCH FOR
`PREDICATE 8G
`
`aLos,
`SEARCH MAB
`GRUECT NO:
`
`aay,
`LEVEL TBRENO TO
`DISBLAY
`What are yuu
`igukingfor?
`What are you
`isokin:
`far?
`
`What pereysneer
`gapeciy?
`What pasgenger
`cauacity ?
`
`What ere yng
`
`husking for?
`
`
`
`FRG. ZOA
`
`FIG. 298
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 16 of 20
`
`US 9,218,530 B2
`
`FIG. 30
`
`
`Hastrice $24,500 -
`HasSaterPhone 505-555.1234
`
`PREDTHCATE
`
`OBJECT
`
`2007 Toyats Tacoma
`
`#007 Toyota Tacoma
`
`fdain Sublect, Predicate
`and Olgect Tables
`
`2007 Toyota Tacoma
`Hastxterion®nfor White
`¥ iPg
`2507 Toyota Tacoma
`HasPassenger Capacity &
`f
`
`2007 Toydte Tacana
`HasGrassVeh cieWinight 2-3 tons
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 17 of 20
`
`US 9,218,530 B2
`
`FIG. 31
`
`19
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 18 of 20
`
`US 9,218,530 B2
`
` | FIG. 32
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 19 of 20
`
`US 9,218,530 B2
`
`rE“Ou
`
`CeeO
`
`£EDid
`
`21
`
`21
`
`
`
`

`

`U.S. Patent
`
`Dec. 22, 2015
`
`Sheet 20 of 20
`
`US 9,218,530 B2
`
`
`
`FIG. 35
`
`© ronennnnnnn
`
`FIG, 36
`
`22
`
`22
`
`

`

`US 9,218,530 B2
`
`1
`SMARTPHONE-BASED METHODS AND
`SYSTEMS
`
`RELATED APPLICATION DATA
`
`This application claims priority benefit to the following
`provisional patent applications:
`61/410,217, filed Nov. 4, 2010;
`61/449,529, filed Mar. 4, 2011;
`61/467,862, filed Mar. 25, 2011;
`61/471,651, filed Apr. 4, 2011;
`61/479,323, filed Apr. 26, 2011;
`61/483,555, filed May 6, 2011;
`61/485,888, filed May 13, 2011; and
`61/501,602, filed Jun. 27, 2011.
`This application is also a continuation-in-part of applica-
`tion Ser. No. 13/174,258, filed Jun. 30, 2011 (now USS. Pat.
`No. 8,831,279).
`The disclosures ofthese previous applicationsare incorpo-
`rated herein by reference, in their entireties.
`
`20
`
`TECHNICAL FIELD
`
`The present technology primarily concerns consumerelec-
`tronic devices, such as smartphones andtablet computers.
`
`25
`
`INTRODUCTION
`
`2
`instances a media-related app may operate in synchrony with
`the audio or video content, e.g., presenting content and links
`at time- or event-appropriate points during the content.
`Apps are now being specialized to particular broadcast and
`recorded media content. The ABC television show My Gen-
`eration, for example, was introduced with a companion iPad
`app dedicated exclusively to the program—providing polls,
`quizzes and other information in synchronized fashion. Tra-
`ditional media companies, such as CNN, ESPN, CBS, etc.,
`are increasingly becoming app companiesas well.
`It is difficult for apps to gain traction in this crowded
`marketplace. Searching iTunes, and other app stores, is the
`most common technique by which users find new apps for
`their devices. The next most popular technique for app dis-
`covery is through recommendations from friends. Both
`approaches, however, were established when the app market
`was much smaller, and have not scaled well.
`In the case of the My Generation iPad ap, for example, the
`show’s producers must reach out to the target audience and
`entice them to go to the app store, where they must type in the
`title ofthe app, downloadit, install it, and then run it when the
`television program is playing.
`In accordance with certain embodiments of the present
`technology, a different solution is provided. In one such
`embodiment, a microphone-equipped user device samples
`ambient content, and produces content-identifying data from
`the captured audio. This content-identifying data is then used
`to look-up an app recommended by the proprietor of the
`content, which app is then installed and launched—withlittle
`or no action required by the user.
`By such arrangement, the content effectively selects the
`app. The user doesn’t select the software; the user’s activity
`selects the software. Over time, each user device becomes
`app-adapted to the content preferences of the user—thereby
`becoming optimized to the user’s particular interests in the
`content world.
`
`To some degree, this aspect of the present technologyis
`akin to the recommendation features of TiVo, but for apps.
`The user’s content consumption habits (and optionally those
`of the user’s social network friends) lead the device to rec-
`ommend appsthat serve the user’s interests.
`Desirably,it is artists that are given the privilege of speci-
`fying the app(s) to be invokedbytheir creative works. Many
`countries have lawsthat recognize artists’ continuing interest
`in the integrity with which their worksare treated (so-called
`“moral rights”). Embodiments of the present technology
`serve this interest—providingartists a continuing role in how
`their art is presented, enabling them to prescribe thepreferred
`mechanisms by which their works are to be experienced.
`Continuity is provided betweenthe artist’s intention and the
`art’s delivery.
`It is not just stand-alone apps that can be treated in this
`fashion. More granular software choices can similarly be
`made, such as the selection of particular rendering codecs to
`be used by media players (e.g., Windows Media Player). For
`example, the National Hockey League may prefer that its
`content be rendered with a codec designed for maximum
`framerate. In contrast, the Food Network mayprefer thatits
`content be rendered with a codec optimized for truest color
`fidelity.
`Historically, the “channel” was king, and content played a
`supporting role (1.e., drawing consumers to the channel, and
`to its advertising). From the consumer’s standpoint, however,
`these roles should be reversed: content should be primary.
`Embodiments of the present technology are based on this
`premise. The user chooses the content, and the delivery
`mechanism then follows, as a consequence.
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`23
`
`filed Apr. 15, 2011 (published as
`
`filed Apr. 4, 2011 (published as
`
`filed Jun. 9, 2010 (published as
`
`filed Mar. 3, 2010 (published as
`
`filed Jun. 24, 2009 (published as
`
`filed Jan. 8, 2007 (published as
`
`Thepresent technology builds on work detailed in previous
`patent filings. These include applications:
`Ser. No. 13/149,334, filed May 31, 2011 (now USS. Pat. No.
`8,842,875);
`Ser. No. 13/088,259,
`20120218444);
`Ser. No. 13/079,327,
`20120046071);
`Ser. No. 13/011,618, filed Jan. 21, 2011 (now US'S. Pat. No.
`8,805,110);
`Ser. No. 12/797,503,
`20110161076);
`Ser. No. 12/774,512, filed May 5, 2010 (U.S. Pat. No. 8,401,
`224);
`Ser. No. 12/716,908,
`20100228632);
`Ser. No. 12/490,980,
`20100205628);
`Ser. No. 12/271,772, filed Nov. 14, 2008 (published as
`20100119208);
`Ser. No. 11/620,999,
`20070185840);
`US. Pat. No. 7,003,731; and
`US. Pat. No. 6,947,571.
`In the few years since their introduction, portable comput-
`ing devices (e.g., smartphones, music players, and tablet
`computers) havetransitioned from novelties to near-necessi-
`ties. With their widespread adoption has comean explosion in
`the numberof software programs (“apps”) available for such
`platforms. Over 300,000 apps are now available from the
`Apple iTunesstore alone.
`Manyapps concern media content. Some are designed to
`provide on-demand playback of audio or video content, e.g.,
`television shows. Others serve to complement media content,
`such as by enabling access to extra content (behind-the-
`scenes clips, cast biographies and interviews, contests,
`games, recipes, how-to videos), by allowing social network-
`based features (communicating with other fans, including by
`Twitter, Facebook and Foursquare, blogs), etc.
`In some
`
`23
`
`

`

`US 9,218,530 B2
`
`3
`The foregoing and other features and advantages of the
`present technology will be more readily apparent from the
`following detailed description, which proceeds with refer-
`ence to the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a system that can be used in
`certain embodiments of the present technology.
`FIG. 2 is a representation of a data structure that can be
`used with the embodiment of FIG. 1.
`
`FIGS. 3-7 detail features of illustrative gaze-tracking
`embodiments, e.g., for text entry.
`FIGS. 8 and 9 detail features of an illustrative user inter-
`face.
`
`FIG. 10 shows a block diagram of a system incorporating
`principles of the present technology.
`FIG. 11 shows marker signals in a spatial-frequency
`domain.
`
`FIG. 12 shows a mixed-domain view ofa printed object
`that includes the markersignals of FIG. 11, according to one
`aspect of the present technology.
`FIG. 13 shows a corner markerthat can be usedto indicate
`hidden data.
`
`FIG. 14 showsanalternative to the marker signals of FIG.
`11.
`
`FIG. 15 showsa graph representation of data output from a
`smartphone camera.
`FIG. 16 shows a middleware architecture for object recog-
`nition.
`
`FIG.17 is similar to FIG. 16, butis particular to the Digi-
`marc Discover implementation.
`FIG. 18 is a bar chart showing impact of reading image
`watermarks on system tasks.
`FIG. 19 further details performance of a watermark recog-
`nition agent running on an Apple iPhone 4 device.
`FIG. 20 showslocationsof salientpoints in first and second
`image frames.
`FIG. 21 showshistogramsassociated with geometric align-
`ment of two framesofsalient points.
`FIG. 22 shows an image memory in a smartphone,includ-
`ing three color bit plane of 8-bit depth each.
`FIG. 23 shows a similar smartphone memory, but now
`utilized to store RDFtriples.
`FIG. 24 shows someof the hundreds or thousands of RDF
`
`triples that may be stored in the memory of FIG. 23.
`FIG. 25 showsthe memory of FIG. 23, now populated with
`illustrative RDF information detailing certain relationships
`among people.
`FIG. 26 shows someofthe templates that may be applied to
`the Predicate plane ofthe FIG. 25 memory, to perform seman-
`tic reasoning on the depicted RDFtriples.
`FIG. 27 names the nine RDFtriples within a 3x3 pixel
`block of memory.
`FIG. 28 showsa store of memory in a smartphone.
`FIGS. 29A and 29B depict elements of a graphical user
`interface that uses data from the FIG. 28 memory.
`FIG. 30 showsuse of a memory storing triples, and asso-
`ciated tables, to generate data used in generate a search query
`report to a user.
`FIG. 31 showsanother store of memory in a smartphone,
`depicting four of more planesofinteger(e.g., 8-bit) storage.
`FIG.32illustrates that a distinctive graphical effect may be
`presented on a device display to indicate the presence of a
`steganographic digital watermark.
`
`4
`FIG.33 illustrates that detection ofa steganographicdigital
`watermark can spawn plural user-tappable tags on a device
`touchscreen.
`
`FIG. 34 showshowthe tags of FIG. 33 can remain fixed in
`position, even as captured imagery changesin position.
`FIG. 35 showsobject handles, to which the tags like that
`shown in FIG. 33 can be tethered.
`
`FIG. 36 shows how plural different distinctive graphical
`effects can be presented, when plural different stegano-
`graphic digital watermarksare detected.
`
`DETAILED DESCRIPTION
`
`The present technology, in somerespects, expands on tech-
`nology detailed in the assignee’ s above-detailed patent appli-
`cations. The reader is presumed to be familiar with such
`previous work, which can be used in implementations of the
`present technology (and into which the present technology
`can be incorporated).
`Referring to FIG. 1, an illustrative system 12 includes a
`device 14 having a processor 16, a memory 18, one or more
`input peripherals 20, and one or more output peripherals 22.
`System 12 mayalso include a network connection 24, and one
`or more remote computers 26.
`An illustrative device 14 is a smartphoneor a tablet com-
`puter, although any other consumerelectronic device can be
`used. The processor can comprise a microprocessor such as
`an Atom or A4 device. The processor’s operation is con-
`trolled,in part, by information stored in the memory, such as
`operating system software, application software (e.g.,
`“apps”), data, etc. The memory may comprise flash memory,
`a hard drive, etc.
`The input peripherals 20 may include a camera and/or a
`microphone. The peripherals (or device 14 itself) may also
`comprise an interface system by which analog signals
`sampled by the camera/microphoneare convertedinto digital
`data suitable for processing by the system. Other input
`peripherals can include a touch screen, keyboard, etc. The
`output peripherals 22 can include a display screen, speaker,
`etc.
`
`The network connection 24 can be wired (e.g., Ethernet,
`etc.), wireless (WiFi, 4G, Bluetooth,etc.), or both.
`In an exemplary operation, device 14 receives a set of
`digital content data, such as through a microphone 20 and
`interface, through the network connection 24, or otherwise.
`The content data may be of any type; audio is exemplary.
`The system 12 processesthe digital content data to gener-
`ate corresponding identification data. This may be done, e.g.,
`by applying a digital watermark decoding process, or a fin-
`gerprinting algorithm—desirably to data representing the
`sonic or visual information itself, rather than to so-called
`“out-of-band”data (e.g., file names, header data, etc.). The
`resulting identification data serves to distinguish the received
`content data from other data of the same type (e.g., other
`audio or other video).
`Byreference to this identification data, the system deter-
`mines corresponding software that should be invoked. One
`way to do this is by indexing a table, database, or other data
`structure with the identification data, to thereby obtain infor-
`mation identifying the appropriate software. An illustrative
`table is shown conceptually in FIG. 2.
`In someinstancesthe data structure may return identifica-
`tion ofa single software program.In that case, this software is
`launched—favailable. (Availability does not require that the
`software be resident on the device. Cloud-based apps may be
`available.) If not available, the software may be downloaded
`(e.g., from an online repository, such as the iTunesstore),
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`24
`
`24
`
`

`

`US 9,218,530 B2
`
`5
`installed, and launched. (Or, the device can subscribe to a
`software-as-service cloud version ofthe app.) Involvement of
`the user in such action(s) can depend on the particular imple-
`mentation: sometimes the user is asked for permission; in
`other implementations such actions proceed without disturb-
`ing the user.
`Sometimesthe data structure mayidentify several different
`software programs. The different programs maybespecific to
`different platforms, in which case, device 12 may simply pick
`the program corresponding to that platform (e.g., Android
`G2, iPhone4, etc.). Or, the data structure may identify several
`alternative programsthat can be used on a given platform. In
`this circumstance,
`the device may check to determine
`which—if any—isalready installed and available. If such a
`program is found, it can be launched. If two such programs
`are found, the device may choose between them using an
`algorithm (e.g., most-recently-used; smallest memory foot-
`print; etc.), or the device may promptthe userfor a selection.
`If noneofthe alternative programsis available to the device,
`the device can select and download one—again using an
`algorithm, or based on input from the user. Once downloaded
`and installed, the application is launched.
`(Sometimesthe data structure may identify different pro-
`gramsthat serve different functions—all related to the con-
`tent. One, for example, may be an app for discovery of song
`lyrics. Another maybe an apprelating to musician biography.
`Another may be an app for purchase of the content. Again,
`each different class of software may include several alterna-
`tives.)
`Note that the device may already have an installed appli-
`cation that is technically suited to work with the received
`content(e.g., to render an MPEG4or an MP3file). For certain
`types of operations, there may be dozens or more such pro-
`gramsthat are technically suitable. However, the content may
`indicate that only a subset of this universe of possible soft-
`ware programs should be used.
`Software in the device 14 maystrictly enforce the content-
`identified software selection. Alternatively, the system may
`treat such software identification as a preferencethat the user
`can override. In some implementations the user may be
`offered an incentive to use the content-identified software. Or,
`conversely, the user may be assessed a fee, or other impedi-
`ment, in order to use software other than that indicated by the
`content.
`
`6
`One formaldefinition of “context”is “any information that
`can be used to characterize the situation ofan entity (a person,
`place or object that is considered relevant to the interaction
`between a user and an application, including the user and
`applications themselves.”
`Context information can be of manysorts, including com-
`puting context (network connectivity, memory availability,
`processor type, CPU contention,etc.), user context (user pro-
`file, location, actions, preferences, nearby friends, social net-
`work(s) andsituation, etc.), physical context (e.g., lighting,
`noise level, traffic, etc.), temporal context (time of day, day,
`month, season,etc.), history of the above, etc.
`In the illustrated table, rows 32 and 34 correspondto the
`same content(i.e., same content ID), but they indicate differ-
`ent software should be used—depending on whether the
`user’s context is indoors or outdoors. (The software is indi-
`cated by a 5 symbolhexidentifier; the content is identified by
`6 hex symbols. Identifiers of other forms, and longer or
`shorter in length, can of course be used.)
`Row 36 showsa software selection that includes two items
`of software—both of which are invoked. (One includes a
`further descriptor—anidentifier of a YouTubevideothatis to
`be loaded by software “FF245-’) This software is indicated
`for a userin a daytime context, and for a user in the 20-25 age
`demographic.
`Row 38 shows user location (zip code) and gender as
`contextual data. The software for this content/context is
`specified in the alternative (ie.,
`four identifiers “OR’d
`together, as contrasted with the “AND”of row 36).
`Rows 40 and 42 show that the same content ID can corre-
`
`spond to different codecs—depending on the device proces-
`sor (Atom or A4).
`(By point of comparison, consider the procedure by which
`codecs are presently chosen. Typically the user isn’t familiar
`with technical distinctions between competing codecs, and
`the artist has no say. Codec selection is thus made by neither
`party that is most vitally interested in the choice. Instead,
`default codecs come bundled with certain media rendering
`software (e.g., Windows Media Player). If the defaults are
`unable to handle certain content, the rendering software typi-
`cally downloads a further codec—again with no input from
`the parties most concerned.)
`It will be understoodthat the software indicatedin table 30
`
`20
`
`25
`
`30
`
`35
`
`40
`
`by the content can be a stand-alone app, or a software com-
`ponent—suchas a codec, driver, etc. The software can render
`Sometimes the system may decline to rendercertain con-
`the content, or it can be a content companion—providing
`tent on a device (e.g., because of lack of suitable app or
`other information or functionality related to the content. In
`hardware capability), but may invite the user to transfer the
`some implementations the “software” can comprise a URL,
`content to another user device that has the needed capability,
`or other data/parameter that is provided to another software
`and may implementsuch transfer. (Ansel Adams might have
`program or online service (e.g., a YouTube video identifier).
`taken a dim view ofhis large format photography being used
`as a screen saver on a small format, low resolution, smart-
`Desirably,all such software identified in the table is chosen
`by the proprietor(e.g., artist, creator or copyright-holder) of
`phonedisplay. If such display is attempted, the software may
`the content with whichit is associated. This affords the pro-
`invite the userto insteadtransfer the imagery to a large format
`prietor a measure of artistic control that is missing in most
`HDdisplay at the user’s homefor viewing.)
`other digital content systems. (The proprietor’s control in
`Instead of absolutely declining to render the content, the
`system may render it in a limited fashion. For example, a
`such matters should be given more deference than,say, that of
`a content distributor—such as AOL or iTunes. Likewise, the
`video might be renderedasaseries of still key frames(e.g.,
`from scenetransitions). Again, the system can transfer the
`proprietor’s choice seems to merit more weight than that of
`content where it can be more properly enjoyed, or—if hard-
`the company providing word processing and spreadsheet
`ware considerations permit (e.g., screen display resolution is
`software for the device.)
`adequate)—needed software can be downloaded and used.
`Often the proprietor’s selection of software will be based
`on aesthetics and technical merit. Sometimes, however, com-
`As shownbythe table of FIG. 2 (which data structure may
`be resident in the memory 18, or ina remote computer system
`mercial considerations comeintoplay. (Asartist Robert Genn
`26), the indication of software may be based on one or more
`noted,“Starving artist’ is acceptable at age 20, suspect at age
`contextual factors—in addition to the content identification
`40, and problematical at age 60.”)
`data. (Only two context factors are shown; more orless can of
`Thus, for example, ifa user’s device detects ambient audio
`course be used.)
`by the group The Decemberists, artist-specified data in the
`
`45
`
`50
`
`55
`
`60
`
`65
`
`25
`
`25
`
`

`

`US 9,218,530 B2
`
`7
`data structure 30 mayindicate that the device should load the
`Amazon app for purchase of the detected music (or load the
`corresponding Amazon web page), to induce sales. If the
`same device detects ambient audio by the Red Hot Chili
`Peppers, that group may have specified that the device should
`load the band’s own web page(or another app), for the same
`purpose. The proprietor can thus specify the fulfillment ser-
`vice for content objected-oriented commerce.
`In somearrangements, the starving artist problem maybest
`beredressed by an auction arrangement. Thatis, the device 14
`(or remote computer system 26) may announce to an online
`service (akin to Google AdWords)that the iPod ofa user—for
`which certain demographic profile/context information may
`be available—has detected the soundtrack of the movie Ava-
`
`tar. A mini auction can then ensue—forthe privilege of pre-
`senting a buying opportunity to the user. The winner(e.g.,
`EBay) then pays the winning bid amount into an account,
`from whichit is shared with the auction service,theartist, etc.
`The user’s device responds by launching an EBay app
`through which the user can buy a copy of the movie, its
`soundtrack, or related merchandise. Pushing such content
`detection events, and associated context
`information,
`to
`cloud-based services can enable a richly competitive market-
`place of responses.
`(Auction technologyis also detailed in the assignee’s pre-
`viously-cited patent applications, and in Google’s published
`patent applications US2010017298 and US2009198607.)
`The popularity of content can lead associated software to
`becomesimilarly popular. This can induce other content pro-
`prietors to consider such software for use with their own
`content, since wide deploymentof that software mayfacili-
`tate consumer exposure to the other proprietor’s content.
`For example, Universal Music Group may digitally water-
`markall its songs with an identifier that causes the FFmpeg
`MP3player to be identified