Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 1 of 68 Page ID
`#:16606
`
`
`
`
`
`
`EXHIBIT 1
`
`
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 2 of 68 Page ID
`#:16607
`I 1111111111111111 11111 111111111111111 1111111111 1111111111 1111111111 11111111
`US006947571Bl
`
`(12) United States Patent
`Rhoads et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,947,571 Bl
`Sep.20,2005
`
`(54) CELL PHONES WITH OPTICAL
`CAPABILITIES, AND RELATED
`APPLICATIONS
`
`(75)
`
`Inventors: Geoffrey B. Rhoads, West Linn, OR
`(US); Megan I. Livermore, Portland,
`OR (US)
`
`(73) Assignee: Digimarc Corporation, Beaverton, OR
`(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/571,422
`
`(22) Filed:
`
`May 15, 2000
`
`(63)
`
`(60)
`
`(51)
`(52)
`(58)
`
`(56)
`
`Related U.S. Application Data
`
`Continuation-in-part of application No. 09/552,998, filed on
`Apr. 19, 2000, now abandoned, and a continuation-in-part of
`application No. 09/547,664, filed on Apr. 12, 2000, and a
`continuation-in-part of application No. 09/543,125, filed on
`Apr. 5, 2000, and a continuation-in-part of application No.
`09/531,076, filed on Mar. 18, 2000, and a continuation-in(cid:173)
`part of application No. 09/343,104, filed on Jun. 29, 1999,
`now abandoned, and a continuation-in-part of application
`No. 09/343,101, filed on Jun. 29, 1999, now abandoned, and
`a continuation-in-part of application No. 09/342,971, filed
`on Jun. 29, 1999, now abandoned, and a continuation-in-part
`of application No. 09/342,689, filed on Jun. 29, 1999, now
`Pat. No. 6,311,214, and a continuation-in-part of application
`No. 09/342,688, filed on Jun. 29, 1999, now Pat. No.
`6,650,761.
`Provisional application No. 60/164,619, filed on Nov. 10,
`1999, provisional application No. 60/163,332, filed on Nov.
`3, 1999, provisional application No. 60/158,015, filed on
`Oct. 6, 1999, provisional application No. 60/151,586, filed
`on Aug. 30, 1999, and provisional application No. 60/141,
`468, filed on Jun. 29, 1999.
`Int. Cl.7 .................................................. H04K 1/00
`U.S. Cl. ..................................... 382/100; 455/566.1
`Field of Search ................................. 382/100, 232;
`380/210, 287, 54; 713/176; 455/556.1,
`557; 348/14.02; 379/900, 910
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,450,490 A
`5,673,316 A
`5,742,845 A
`5,761,686 A
`5,778,102 A
`5,857,038 A
`5,892,900 A
`5,893,037 A
`5,900,608 A
`5,915,027 A
`
`9/1995 Jensen et al. .................. 380/6
`9/1997 Auerbach et al. .. ... ... ... ... 380/4
`4/1998 Wagner ...................... 395/831
`6/1998 Bloomberg ................. 707 /529
`7/1998 Sandford, II et al. ....... 382/251
`1/1999 Owada et al.
`.............. 382/254
`4/1999 Ginter et al.
`............... 395/186
`4/1999 Reele et al.
`................ 455/556
`5/1999 Iida ............................ 235/381
`6/1999 Cox et al. ..................... 380/54
`
`6,052,486 A
`6,072,494 A
`6,115,482 A
`6,164,534 A
`6,192,257 Bl
`6,195,104 Bl
`6,252,598 Bl
`6,317,609 Bl
`6,408,331 Bl
`6,573,883 Bl
`2001/0001854 Al
`2001/0051996 Al
`2002/0034297 Al
`2002/0064297 A 1
`2002/0075298 Al
`2002/0122568 Al
`2003/0011684 Al
`
`. . . . . . . . . . 382/232
`Knowlton et al.
`Nguyen ...................... 345/358
`Sears et al. . . . . . . . . . . . . . . . . . 382/114
`Rathus et al. ............... 235/380
`Ray
`........................... 455/566
`Lyons
`........................ 345/473
`Segen ........................ 345/358
`Alperovich et al.
`........ 455/556
`Rhoads ....................... 709/217
`Bartlett ....................... 345/156
`Schena et al.
`................ 705/27
`Cooper et al. .............. 709/217
`Rhoads ....................... 380/247
`Brunk et al. ................ 382/100
`Schena et al.
`.............. 345/738
`Zhao .......................... 382/100
`Narayanaswami et al. .. ... .. 348/
`207.99
`2003/0141375 Al * 7/2003 Lawandy .................... 235/494
`
`4/2000
`6/2000
`9/2000
`* 12/2000
`2/2001
`2/2001
`6/2001
`11/2001
`6/2002
`6/2003
`5/2001
`12/2001
`3/2002
`5/2002
`6/2002
`9/2002
`1/2003
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`WO
`
`1152592 Al
`W000/36605 Al
`
`11/2001
`6/2000
`
`H04N/1/32
`........... GllB/23/28
`
`OTHER PUBLICATIONS
`
`Rekimoto, "Titling Operations for Small Screen Interfaces
`(Tech Nole)," ACM Symp. on User Interface Software anc.l
`Technology, 1996.
`Fitzmaurice, "Situated Information Spaces and Spatially
`Aware Palmtop Computers," Comm. of the ACM, vol. 36,
`No. 7, pp. 39-49, Jul. 1993.
`Rekimoto, "The Word through the Computer: Computer
`Augmented Interaction with Real World Environments,"
`ACM Symp. on User Interface Software and Technology,
`1995.
`Small et al, "Design of Spatially Aware Graspable Dis(cid:173)
`plays," Extended Abstracts of CHI '97, 2 pp.
`
`* cited by examiner
`
`Primary Examiner-Andrew W. Johns
`(74) Attorney, Agent, or Firm----Digimarc Corporation
`
`(57)
`
`ABSTRACT
`
`A cell phone is equipped with a 2D optical sensor, enabling
`a variety of applications. For example, such a phone may
`also be provided with a digital watermark decoder, permit(cid:173)
`ting decoding of steganographic data on imaged objects.
`Movement of a phone may be inferred by sensing movement
`of an imaged pattern across the optical sensor's field of view,
`allowing use of the phone as a gestural input device through
`which a user can signal instructions to a computer-based
`process. A variety of other arrangements by which electronic
`devices can interact with the physical world are also
`detailed, e.g., involving sensing and responding to digital
`watermarks, bar codes, RFIDs, etc.
`
`4 Claims, 16 Drawing Sheets
`
`BOFA00000400
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 3 of 68 Page ID
`#:16608
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 1 of 16
`
`US 6,947,571 Bl
`
`FIG. 1
`
`BOFA00000401
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 4 of 68 Page ID
`#:16609
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 2 of 16
`
`US 6,947,571 Bl
`
`0
`..-
`
`\
`
`0
`co
`.....
`
`('I')
`:it:
`w
`(..)
`0::
`::J
`0
`en
`w
`a:::
`
`N
`~
`w
`(..)
`c::
`:::::,
`0
`en
`w c::
`
`.....
`~
`w
`0
`0::
`:::,
`0
`en
`w
`0::
`
`0)
`..-
`
`z
`0
`i= 0::
`c2 ~
`I- c::
`en w
`(!) en
`w
`c::
`
`r--
`.....
`
`co
`.....
`
`z
`Ow
`1-cn
`<( <(
`0:: CD
`I- c:(
`en I-
`-c:(
`(!) 0
`w
`c::
`
`-
`
`w
`I- a::: 0
`(..) w LL.
`:::::, _J LL.
`0 0 0
`oz~
`c:: <( 0
`a. J: <(
`E.
`
`.....
`'o:t
`
`.....
`LO
`
`(!)
`0
`_J
`
`c2 0::
`WW
`I- >
`::J c::
`ow
`c:: Cl)
`
`N
`•
`
`(!) -LL
`
`0
`N
`
`·o
`!;.-
`I- C UJ
`0 <( u.i
`w a_,
`-, w rr::
`aP-<
`O z_
`-o
`o:w
`~::E
`
`.....
`N
`
`(!)
`z
`-w
`I- 0
`c:(_
`Z>
`-w
`(!) CJ
`0::
`0
`
`0::
`I- w
`o-
`W LL
`-, I-
`mz
`ow
`0
`
`BOFA00000402
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 5 of 68 Page ID
`#:16610
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 3 of 16
`
`US 6,947,571 Bl
`
`NETWORK
`
`ORIGINATING DEVICE
`28a
`
`28b
`
`28c
`
`APPLICATION #1
`
`APPLICATION #2
`(BROWSER)
`
`MEDIABRIDGE
`APPLICATION
`WATERMARK
`DECODER
`
`DEVICE DRIVER
`[ WATERMARK i
`! DECODER
`! 26
`', 30
`
`INTERFACE ELECTRONICS
`
`SENSOR
`
`22
`
`12
`
`FIG. 3
`
`BOFA00000403
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 6 of 68 Page ID
`#:16611
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 4 of 16
`
`US 6,947,571 Bl
`
`Customers and Products - - - - . . . . J
`
`Business Support
`Systems
`
`Registrant Info +
`Action(s)
`
`n
`Database(s
`
`i~bf
`
`ML page.
`msgs,
`ddress
`
`Monitoring
`
`P
`
`uct Reports
`
`FIG. 4
`
`BOFA00000404
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 7 of 68 Page ID
`#:16612
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 5 of 16
`
`US 6,947,571 Bl
`
`Customers and Product~~,;~~~ Grand Central
`-ffi'lirf~ciJ?.'. (Transaction Router)
`:ii.1:r~:..
`
`Supported Vendors
`and Products
`
`"Reply to• Addres +
`Decoded requ st
`
`Product
`Handlers
`
`FIG. 5
`
`BOFA00000405
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 8 of 68 Page ID
`#:16613
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 6 of 16
`
`US 6,947,571 Bl
`
`Customers and Products
`
`Grand Central
`{Product Registration)
`
`Serial
`
`ser
`Registration
`Session
`
`ser
`Registration
`Session
`
`ser
`Registration
`Session
`
`ser
`Registration
`Session
`
`ontracl
`Info
`
`Action a dit log
`
`FIG. 6
`
`BOFA00000406
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 9 of 68 Page ID
`#:16614
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 7 of 16
`
`US 6,947,571 Bl
`
`Req sts fDI' eports
`
`Transaction
`Log
`
`Registra~on
`Database
`
`FIG. 7
`
`BOFA00000407
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 10 of 68 Page ID
`#:16615
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 8 of 16
`
`US 6,947,571 Bl
`
`~
`~ h
`~
`
`-
`.........
`-
`--
`-·-
`~--
`
`__,..
`~
`J...-_ ..
`
`... ,_
`
`Bl IFSFlt
`lQ c - •
`:IQ c - •
`' !Q t - •
`nc:-•
`lQI sir . . .
`
`'1111,00.alM
`J/17QltSl'lf
`Jll11UDUIFlf
`J117All.41PM
`Jll11UD:12!PM
`:1111.WJ;ztPM
`J/111UDHll'M
`J111IID41f PM
`
`IQ __ ...._
`'IQ - - - -
`
`WIIIHllll'M
`IQ _ 1 1 ,~ - - 11111UDet:il'M
`
`,.. ---- l/17AJD',t11Plt
`lll ---- :IIIMIH11fM
`--- :1'111UDl2tl'N
`. , .. _ l/17AIIDN
`.. 0Pff1t
`UI -- :IIIMIIDN
`... -- :11111UDD,..
`.. _,_ 311MIOd,..
`--- 3'1145:nN
`
`VUII ,.J•
`Bl tDn.
`
`SIZl!lll:r:IHII
`l/111118.SN
`
`SI _,..
`
`l/17.W tll/l'M
`311MIIIZIN
`
`:D -11114\,0.... l/11111115,0N
`
`IQ-lll~O.C... :IIIMI0511 l'N
`:n a.wn.
`311MIO!l1Jfll
`
`,.
`...
`...
`....
`A
`...
`...
`...
`,.
`
`A
`A
`M
`A
`...
`...
`A
`...
`...
`...
`..
`...
`...
`
`A
`
`A
`
`FIG. 8
`
`BOFA00000408
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 11 of 68 Page ID
`#:16616
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 9 of 16
`
`US 6,947,571 Bl
`
`---.....
`·-(cid:173)...... _,..
`...._._
`...........,. _ ...
`~ ·-(cid:173)-.........,_._
`..........
`
`FIG. 9
`
`BOFA00000409
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 12 of 68 Page ID
`#:16617
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 10 of 16
`
`US 6,947,571 Bl
`
`FIG. 10
`
`BOFA00000410
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 13 of 68 Page ID
`#:16618
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 11 of 16
`
`US 6,947,571 Bl
`
`110
`
`\
`
`NETWORK
`
`122
`
`~ COMPUTER
`
`r 1 - -~ :--M~~;~;-7 ._ __ _J116
`
`112
`
`---------• 114
`'124
`
`DISPLAY
`
`FIG. 11
`
`BOFA00000411
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 14 of 68 Page ID
`#:16619
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 12 of 16
`
`US 6,947,571 Bl
`
`FIG. 12
`
`r124
`COMPUTER
`
`►:
`I
`130
`
`128
`I
`
`, _____ j _____ ~
`I OTHER
`:
`:
`IMAGING
`:PROGRAMS:
`------------'
`
`I
`
`BEDOOP
`DECODER
`'-132
`
`l
`------ -----.
`.- ~ BE DOOP AP I
`: - - - - - -r - - - - -·
`
`I
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-1
`
`1
`
`134
`\
`140
`~ -: BEDOOP AP•
`I
`BEDOOP -- ~ - - - - - -f - - - - _, 136
`REGISTRY
`~-f BEDOOP AP
`BROWSER'
`o.s.
`&
`: - - - - - J - - - - -,
`I
`133-1
`-~ BEDOOP AP•
`- ---- -·- --- --·
`I
`I
`I
`I
`I
`I
`I
`
`MASTER
`REGISTRATION
`SERVER
`
`148a
`
`146
`
`,..-148c l ,,-----148b J\ I
`. . . SERVER SERVER SERVER . . . SERVER ...
`
`CLASS
`
`CLASS
`
`CLASS
`
`CLASS
`
`150........._
`
`. . . DNS
`
`I
`
`I
`
`r
`
`l
`l
`DNS . . . DNS ...
`
`DNS
`I
`
`l
`. . . DNS
`
`l
`DNS
`
`1
`DNS . . . DNS ...
`
`158
`, I
`FURTHER
`SERVER
`''-\
`159
`
`I
`
`I
`
`I
`I
`
`I
`
`I
`
`I
`156,
`
`I
`
`l
`
`. . .
`
`DNS LEAF
`NODE SERVER
`
`----------
`
`BOFA00000412
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 15 of 68 Page ID
`#:16620
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 13 of 16
`
`US 6,947,571 Bl
`
`AUXILIARY
`DEVICE
`
`FIG. 13
`
`✓ 10
`
`REMOTE
`INTERFACE
`222
`
`CPU
`
`SECOND
`2DCCD
`ARRAY
`218 b
`
`(Prior Art)
`
`U.I. (LCD &
`surroNs)
`
`MEMORY
`
`220
`
`214
`
`FIRST2D
`CCD
`ARRAY
`218a
`
`LINEAR
`CCD
`ARRAY
`216
`
`FIG. 14
`
`BOFA00000413
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 16 of 68 Page ID
`#:16621
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 14 of 16
`
`US 6,947,571 Bl
`
`Grid Orientation Under
`CCD 218a
`
`Grid Orientation Under
`CCD 218b
`
`FIG. 15
`
`FIG. 16
`
`BOFA00000414
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 17 of 68 Page ID
`#:16622
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 15 of 16
`
`US 6,947,571 Bl
`
`.
`(!) -LL
`
`.......
`t::
`.......
`~ <(
`. ~
`- "i:
`(!) 0
`LL D.. -
`
`BOFA00000415
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 18 of 68 Page ID
`#:16623
`
`U.S. Patent
`
`Sep.20,2005
`
`Sheet 16 of 16
`
`US 6,947,571 Bl
`
`CD
`
`T"" .
`(!) -LL
`
`BOFA00000416
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 19 of 68 Page ID
`#:16624
`
`US 6,947,571 Bl
`
`1
`CELL PHONES WITH OPTICAL
`CAPABILITIES, AND RELATED
`APPLICATIONS
`
`RELATED APPLICATION DATA
`
`This application claims priority benefit of each of the
`following provisional applications:
`60/141,468, filed Jun. 29, 1999
`60/151,586, filed Aug. 30, 1999
`60/158,015, filed Oct. 6, 1999
`60/163,332, filed Nov. 3, 1999
`60/164,619, filed Nov. 10, 1999
`This application is also a continuation-in-part of each of the
`following applications
`Ser. No. 09/314,648, filed May 19, 1999, now U.S. Pat.
`No. 6,681,028
`Ser. No. 09/342,688, filed Jun. 29, 1999, now U.S. Pat.
`No. 6,650,761
`Ser. No. 09/342,689, filed Jun. 29, 1999, now U.S. Pat. 20
`No. 6,311,214
`Ser. No. 09/342,971, filed Jun. 29, 1999, now abandoned
`Ser. No. 09/343,101, filed Jun. 29, 1999, now abandoned
`Ser. No. 09/343,104, filed Jun. 29, 1999, now abandoned
`Ser. No. 09/531,076, filed Mar. 18, 2000
`Ser. No. 09/543,125, filed Apr. 5, 2000
`Ser. No. 09/547,664, filed Apr. 12, 2000
`Ser. No. 09/552,998, filed Apr. 19, 2000, now abandoned
`
`45
`
`BOFA00000417
`
`2
`not even be "digital," per se ). Different implementation
`considerations attach to these variants. Likewise, though the
`bulk of this disclosure concentrates on objects which have
`some form of digital message attached thereto, some aspects
`5 of the technology may apply to objects which have no such
`thing, where the prior arts of pattern recognition and gestural
`input can be borrowed in combination with this technology
`to effect yet a broader array of applications.
`Nor, as will be apparent, is the technology limited to
`10 systems employing optical input and encoded imagery.
`Corresponding techniques can also be employed with
`encoded audio. Indeed, any physical or electronic "object"
`can make use of the principles detailed herein.
`"Bedoop." The sound that a refrigerator might make,
`15 outfitted with a simple camera/processor unit/net
`connection, as a ten year old child holds up an empty milk
`carton and a ping goes out to the local grocery store, adding
`the item to an accumulating delivery list. The sound that
`might be heard echoing over and over inside Internet cafes
`as heretofore computerphobes take their first skeptical steps
`onto the World Wide Web. The sound heard at the fast food
`counter as a repeat customer holds up his sandwich card
`ticking off his latest meal, hoping for the sirens to go off for
`a $500 prize given to the lucky customer of the week. Blue
`25 sky scenarios abound.
`Such aspects of the present technology are thus about
`powerful new user interfaces to computers. These new user
`interfaces extend into the everyday world in ways that a
`mouse and keyboard never could. By enabling everyday
`objects to communicate their identities and functions to
`ever-attendant devices, not only will the World Wide Web be
`given an entirely new dimension, but basic home and office
`computing may be in store for some fundamental advances
`as well.
`According to one aspect, the invention includes a method
`of data processing on a computer system, comprising (a)
`using an application program to compose an electronic
`version of a document; (b) printing the document onto paper,
`40 the printing including marking with machine readable indi(cid:173)
`cia encoding plural-bit auxiliary data; and ( c) storing the
`plural-bit auxiliary data in association with data identifying
`a location at which the electronic version of the document is
`stored.
`According to another aspect, the invention includes a
`method of data processing on a computer system, compris(cid:173)
`ing (a) presenting a printed document to an optical capture
`device; (b) processing image data produced by said device
`to decode plural-bit data encoded therein; (c) based on said
`50 decoded plural-bit data, launching a software application
`corresponding,to said printed document; and (d) using said
`software application to open an electronic version of said
`document.
`According to another aspect, the invention includes a
`method of operating a computer, the computer including an
`operating system with a registry database, the registry data(cid:173)
`base associating specific data types with specific software
`programs particularly corresponding thereto, wherein the
`method further includes: (a) providing a frame of image
`data; (b) decoding plural-bit identifier data from the image
`data; ( c) consulting the registry database to identify a
`software program corresponding to said identifier data; and
`(d) invoking the identified software program.
`According to another aspect, the invention includes a
`greeting card having a substrate with visually-perceptible
`indicia printed thereon, wherein the card is encoded with
`plural-bit binary data that can be decoded by an image
`
`30
`
`35
`
`BACKGROUND AND SUMMARY OF THE
`INVENTION
`"Bedoop." That might be the sound someone might hear
`as they lazily place a magazine advertisement in front of
`their desktop camera. Magically, the marketing and sales
`web site associated with the ad is displayed on their com(cid:173)
`puter. More information? Want to buy now? Look at the full
`product line? No problem.
`"Bedoop." That might be the same sound when that same
`someone places their credit card in front of their desktop
`camera. Instantly, the product displayed on the web page is
`purchased. Behind the scenes, a secure purchase link is
`initiated, transmitting all requisite information to the vendor.
`Twist the credit card clockwise and the purchaser chooses
`overnight delivery.
`So goes an exemplary embodiment further described in
`this specifications. Although this example is rather specific,
`it nevertheless alludes to vast array of applications possible
`when an input device such as a digital camera is turned into
`a general-purpose user interface device with an intuitive
`power that very well might rival the mouse and the key(cid:173)
`board.
`One aspect of certain embodiments is that an object or
`paper product so-scanned contains digital information that
`can be quickly read and acted upon by an appropriately 55
`configured device, computer or appliance. Such an embodi(cid:173)
`ment envisions that this digital information is aesthetically
`hidden on objects. These objects have been previously and
`pro-actively marked with the digital information, using any
`of the broad range of data encoding technologies, such as 60
`digital watermarking.
`Although this aspect of the technology concentrates on
`flat object applications wherein the digital information is
`often imperceptibly integrated into the object, it is certainly
`not meant to be so limited. Objects can be three dimensional 65
`in nature and the information more visually overt and/or
`pre-existing (i.e., not "pro-actively" embedded, or it might
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 20 of 68 Page ID
`#:16625
`
`US 6,947,571 Bl
`
`20
`
`25
`
`3
`processing device and used to direct a computer to a web site
`where an image, video, and/or audio presentation corre(cid:173)
`sponding to said card is provided.
`According to another aspect, the invention includes a
`method of providing a customized greeting, comprising: (a)
`providing a greeting card having plural-bit data encoded
`therein; (b) customizing a web site presentation correspond-
`ing to said card; ( c) providing the card to a recipient; ( d)
`decoding the encoded plural-bit data from the card; and (e)
`in response to the decoded plural-bit data, presenting to the 10
`recipient the web site presentation.
`According to another aspect, the invention includes a
`method of printing a magazine, comprising: (a) processing
`an electronic representation of an advertisement to encode
`plural bit data therein; (b) printing a page of advertising in
`accordance with said electronic representation to yield an
`encoded advertisement page; and (c) binding said page into
`a magazine; wherein said plural bit data serves to identify an
`entry in a database, said database entry having an internet
`address of a web page that is associated with said adver(cid:173)
`tisement stored therein.
`According to another aspect, the invention includes a
`promotional method comprising: ( a) encoding a print adver(cid:173)
`tisement to hide plural-bit data therein; (b) processing the
`print advertisement to extract the plural-bit data therefrom;
`and ( c) using at least a part of the extracted plural-bit data
`to direct an internet web browser to a web site that provides
`consumer information related to a product or service pro(cid:173)
`moted by the print advertisement.
`According to another aspect, the invention includes a
`method of determining consumer response to print
`advertising, comprising: ( a) encoding a first print advertise(cid:173)
`ment with first data; (b) encoding a second print advertise(cid:173)
`ment with second data different than the first; ( c) the first and
`second data providing identifiers by which consumer
`devices can link to web pages associated with said adver(cid:173)
`tisements; ( d) monitoring linking traffic due to each of said
`identifiers to thereby determine consumer response to the
`advertisements.
`According to another aspect, the invention includes a
`promotional method comprising: (a) presenting an object
`within the field of view of an optical sensor device, the
`object being selected from the list consisting of a retail
`product, packaging for a retail product, or printed advertis(cid:173)
`ing; (b) acquiring optical data corresponding to the object;
`(c) decoding plural-bit digital data from the optical data; (d)
`submitting at least some of said decoded data to a remote
`computer; and ( e) determining at the remote computer
`whether a prize should be awarded in response to submis(cid:173)
`sion of said decoded data.
`According to another aspect, the invention includes a
`method of interacting with a magazine using a computer, the
`computer including an internet web browser, the method
`including: ( a) providing a peripheral device having a sensor;
`(b) positioning the peripheral device adjacent a first adver(cid:173)
`tisement in the magazine to direct the web browser to a first
`internet address; and ( c) positioning the peripheral device
`adjacent a second advertisement in the magazine to direct
`the web browser to a second internet address.
`According to another aspect, the invention includes a
`computer peripheral and method of its use, the peripheral
`being used in conjunction with a computer system having an
`internet browser associated therewith, the peripheral com(cid:173)
`prising: (a) a housing adapted to fit within a user's palm and
`slide over a medium; (b) an optical sensor having at least one
`sensing element and producing optical data; (c) a lens for
`
`4
`imaging the medium onto the sensor, and the method
`includes: (a) sliding the peripheral over a portion of a printed
`advertisement; (b) processing the optical data to decode
`plural bit information encoded on the advertisement; and (c)
`5 using said plural bit information to direct the internet
`browser to an internet web page associated with said adver(cid:173)
`tisement.
`According to another aspect, the invention includes an
`electronic commerce method comprising: (a) providing a
`printed catalog that includes an image of an article offered
`for sale by a merchant, wherein the image is encoded with
`plural-bit binary data; (b) optically sensing the image to
`produce optical data corresponding thereto; (c) decoding the
`encoded data from the optical data; and ( d) electronically
`ordering the article from the merchant by use of said
`15 decoded data. The ordering may make use of earlier-stored
`customer profile information (e.g., clothing size data), and
`the encoding may be steganographic.
`According to another aspect, the invention comprises a
`wireless telephony handset including a microphone, a
`modulator, and an RF amplifier, the device serving to receive
`audio and transmit an RF signal conveying audio
`modulation, the handset further including an optical sensor
`producing optical data, a lens for imaging an object onto the
`sensor, and a decoder for decoding plural bit identifier data
`conveyed by a barcode or a digital watermark on the object.
`According to another aspect, the invention includes an
`image-based network navigation method permitting a user to
`link to a remote computer, comprising: (a) detecting
`30 encoded data from a printed object; (b) linking to the remote
`computer through a network in accordance with said
`encoded data; and (c) providing the user's zip code to the
`remote computer.
`According to another aspect, the invention includes a
`35 method comprising: (a) sensing an object identifier from a
`first object; (b) sending said first object identifier from a first
`device to a second device; (c) in response, at said second
`device, identifying address information corresponding to
`said first object identifier and sending same to the first
`40 device; ( d) initiating a link from the first device in accor(cid:173)
`dance with said address information; ( e) at said second
`device, identifying additional objects related to said first
`object; identifying additional address information corre(cid:173)
`sponding to said additional objects; and sending said addi-
`45 tional address information to the first device; and (f) storing
`said additional address information in a memory at the first
`device; wherein, if an object included among said identified
`additional objects is sensed by the first device, the corre(cid:173)
`sponding address information can be retrieved from said
`50 memory in the first device without the intervening delays of
`communicating with the second device.
`According to another aspect, the invention includes an
`apparatus having a detector of machine readable data and a
`software program used in conjunction with said machine
`55 readable data, operable to transmit a packet of data to a
`remote system, said packet of data comprising (a) an iden(cid:173)
`tifier of said software program, and (b) at least a portion of
`detected machine readable data.
`According to another aspect, the invention includes an
`60 apparatus having a detector of machine readable data and a
`software program used in conjunction with said machine
`readable data, operable to transmit a packet of data to a
`remote system, said packet of data comprising (a) a context
`or environment identifier, and (b) at least a portion of
`65 detected machine readable data.
`According to another aspect, the invention includes a
`networked computer system, responsive to watermark data
`
`BOFA00000418
`
`

`

`Case 2:20-cv-07872-GW-PVC Document 310-2 Filed 03/28/24 Page 21 of 68 Page ID
`#:16626
`
`US 6,947,571 Bl
`
`5
`sent from a software program on a remote computer, to
`initiate delivery of advertisement data to said remote com(cid:173)
`puter.
`In any of the foregoing arrangements, the encoding can be
`steganographic ( e.g., by digital watermarking), or can
`employ other machine readable data (e.g., barcodes, etc.).
`More generally, the arrangements just reviewed generally
`have counterparts that can be implemented with other than
`optical or image data (e.g., audio data, magnetic stripe
`information, etc.).
`The foregoing just touches on a few of the many inventive
`aspects of the technology detailed below. These and other
`features of the present technology will be more readily
`apparent from the following detailed description, which
`proceeds with reference to the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`15
`
`FIG. 1 is a diagram showing the principal process com(cid:173)
`ponents of an illustrative system employing the present 20
`technology.
`FIG. 2 is a block diagram showing an illustrative system
`for performing the response process of FIG. 1.
`FIG. 3 is a block diagram more particularly detailing an
`originating device used in the system of FIG. 2.
`FIG. 4 illustrates certain top level data flows in the system
`of FIG. 2.
`FIG. 5 illustrates certain data flows associated with the
`router of FIG. 2.
`FIG. 6 illustrates certain data flows associated with the
`registration process of FIG. 2.
`FIG. 7 illustrates certain data flows associated with the
`product handler of FIG. 2.
`FIGS. 8-10 show a sequence of screen shots from an
`illustrative system.
`FIG. 11 is a block diagram showing another embodiment
`of the present technology.
`FIG. 12 is a block diagram showing another embodiment
`of the present technology.
`FIG. 13 is a block diagram of a prior art scanner.
`FIG. 14 shows an object being scanned along an arced
`path.
`FIG. 15 shows how object warping may be detected.
`FIG. 16 shows use of binocular processing to determine
`certain object attributes.
`FIG. 17 shows a watermark grid.
`FIG. 18 shows a pre-warped watermark grid.
`FIG. 19 shows another pre-warped watermark grid.
`
`40
`
`6
`As will be evident, many of the topical sections presented
`below are both founded on, and foundational to, other
`sections. For want of a better rationale, the sections in the
`first part are presented in a more or less random order. It
`5 should be recognized that both the general principles and the
`particular details from each section find application in other
`sections as well.
`Compounding the situation, the present specification
`draws from several priority applications filed over the course
`10 of nearly a year. Accordingly, the same concepts are some(cid:173)
`times expressed several times, each reflecting a different
`perspective-depending on the date and context of the
`application in which it was first found.
`The term "Bedoop" has been superceded in applicant's
`commercialization of the technology by the term Digimarc
`MediaBridge. Both terms are used in this specification but
`refer to the same technology.
`To prevent the length of this disclosure from compound(cid:173)
`ing out of control, the various permutations and combina(cid:173)
`tions of the features of the different sections are not exhaus(cid:173)
`tively detailed. The inventors intend to explicitly leach such
`combinations/permutations, but practicality requires that the
`detailed synthesis be left to those who ultimately implement
`systems in accordance with such teachings.
`Part I
`25 Introduction to Digital Watermarking and other Encoding
`Techniques
`There are nearly as many techniques for digital water(cid:173)
`marking (a type of steganographic data encoding) as there
`are applications for it. The reader is presumed to be familiar
`30 with the great variety of methods. A few are reviewed below.
`The present assignee's prior application Ser. No. 09/127,
`502, filed Jul. 31, 1998, now published as WO0007356,
`shows techniques by which very fine lines can be printed on
`a medium to slightly change the medium's apparent tint,
`35 while also conveying digital data. Commonly-owned appli(cid:173)
`cation Ser. No. 09/074,034, filed May 6, 1998, now pub(cid:173)
`lished as WO 9953428, details how the contours of printed
`imagery can be adjusted to convey digital data. (That
`technique can be applied to printed text characters, as well
`as the line art imagery particularly considered.) The assign(cid:173)
`ee's U.S. Pat. No. 5,850,481 details how the surface of paper
`or other media can be textured to convey optically(cid:173)
`detectable binary data. The assignee's U.S. Pat. Nos. 5,862,
`260, 5,841,886 and 5,809,160 detail various techniques for
`45 steganographically encoding photographs and other imag(cid:173)
`ery.
`Some watermarking techniques arm based on changes
`made in the spatial domain; others are based on changes
`made in transformed domains (e.g., DCT, wavelet).
`Watermarking of printed text can be achieved by slight
`variations to character shape, character kerning, line
`spacing, etc., as shown by various writings by Brassil et al,
`including "Electronic Marking and Identification Tech(cid:173)
`niques to Discourage Document Copying," Proceedings of
`55 INFOCOM '94 Conference on Computer, IEEE Comm. Soc
`Conference, Jun. 12-16, 1994, pp. 1278-1287; "Hiding
`Information in Document Images," Proceedings of the
`Twenty-Ninth Ann