(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2015/0286658 A1
`(43) Pub. Date:
`Oct. 8, 2015
`Folkens et al.
`
`US 20150286658A1
`
`(54)
`(71)
`
`(72)
`
`(73)
`
`(21)
`(22)
`
`(60)
`
`IMAGE PROCESSING
`
`Publication Classification
`
`Applicants: Bradford A. Folkens, Los Angeles, CA
`(US); Dominik K. Mazur, Los Angeles,
`CA (US)
`Inventors: Bradford A. Folkens, Los Angeles, CA
`(US); Dominik K. Mazur, Los Angeles,
`CA (US)
`Assignee: Image Searcher, Inc., Los Angeles, CA
`(US)
`Appl. No.: 14/267,840
`Filed:
`May 1, 2014
`
`Related U.S. Application Data
`Provisional application No. 61/975,691, filed on Apr.
`4, 2014, provisional application No. 61/976,494, filed
`on Apr. 7, 2014, provisional application No. 61/987,
`156, filed on May 1, 2014.
`
`(51) Int. Cl.
`G06F 7/30
`(52) U.S. Cl.
`CPC ...... G06F 17/30268 (2013.01); G06F 17/3028
`(2013.01)
`
`(2006.01)
`
`(57)
`
`ABSTRACT
`
`An image recognition approach employs both computergen
`erated and manual image reviews to generate image tags
`characterizing an image. The computer generated and manual
`image reviews can be performed sequentially or in parallel.
`The generated image tags may be provided to a requester in
`real-time, be used to select an advertisement, and/or be used
`as the basis of an Internet search. In some embodiments
`generated image tags are used as a basis for an upgraded
`image review. A confidence of a computer generated image
`review may be used to determine whether or not to perform a
`manual image review.
`
`
`
`
`
`
`
`Receive
`image
`410
`
`Receive
`Subset
`dentification
`415
`
`Receive
`Source
`Data 420
`
`Determine
`Destination
`465
`
`Post image
`470
`
`Receive
`Analysis
`Priority 425
`
`Queue
`Image 460
`
`Receive
`Evaluation
`
`Distribute
`image 430
`
`Receive
`Automated
`Response
`435
`
`Determine
`Confidence
`440
`
`
`
`
`
`Provide
`Results
`455
`
`445
`
`NO
`
`YES 490
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 1 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 1 of 6
`
`US 2015/0286658A1
`
`
`
`Image Processing System 110
`
`Image
`
`Reviewer
`
`Logic 147
`
`157
`
`Automatic
`identification
`Interface 150
`
`Destination
`Logic
`160
`
`Response
`Logic
`175
`
`Processor
`140
`
`Reviewer
`Pool
`155
`
`image
`Posting
`Logic 1.65
`
`Content
`Processing
`Logic 185
`
`
`
`
`
`Automatic
`ldentification
`System
`152
`
`FIG. 1
`
`N -
`
`-
`
`Advertising
`System
`180
`
`
`
`Destination
`125A
`
`Destination
`125B
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 2 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 2 of 6
`
`US 2015/0286658A1
`
`
`
`FIG. 2
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 3 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 3 of 6
`
`US 2015/0286658A1
`
`ooooo AT&T as
`
`6:50 PM
`
`1 OO %
`
`
`
`-------
`
`The Smooth & Sweet Flavors of Blonde
`Roast Available At Home
`Starbucks.com
`
`---
`
`
`
`f
`
`so
`
`
`
`Breaking Bad TV Series Tra...
`and 4 other related items
`X
`$9.99 - $88.00
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`X
`
`1020 Wilshire Blvd
`Santa Monica, 904O1
`yelps 20 matches found nearby
`starbucks Coffee Cup white eBay
`Visit eBay for great deals on a huge selection starbucks
`Coffee Cup white. Shop eBay.
`www.ebay.corn
`starbucks white coffee cup eBay
`Find great deals on eBay for starbucks white Coffee Cup
`and starbucks replacement lid. Shop with confidence.
`www.ebay.com
`
`FIG. 3
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 4 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 4 of 6
`
`US 2015/0286658 A1
`
`Receive
`Image
`410
`
`Receive
`Subset
`Identification
`415
`
`
`
`
`
`
`
`Receive
`Source
`Data 420
`
`Determine
`Destination
`465
`
`Post image
`470
`
`Receive
`Analysis
`Priority 425
`
`
`
`
`
`Queue
`Image 460
`
`Receive
`Review
`475
`
`Distribute
`image 430
`
`Receive
`Automated
`Response
`435
`
`Determine
`Confidence
`440
`
`NO
`
`
`
`
`
`445
`
`Provide
`Results
`455
`
`
`
`F.G. 4
`
`YES
`
`Upgrade?
`
`NO
`
`490
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 5 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 5 of 6
`
`US 2015/0286658 A1
`
`
`
`Receive
`Image
`410
`
`Receive
`Subset
`identification
`415
`
`Receive
`Source
`Data 420
`
`Determine
`Destination
`465
`
`Post image
`470
`
`
`
`Receive
`Analysis
`Priority 425
`
`Distribute
`Image 430
`
`Receive
`Automated
`Response
`435
`
`Determine
`Confidence
`440
`
`
`
`Receive
`Evaluation
`
`YES
`
`Upgrade? NO
`
`C End)
`
`YES 490
`
`445
`
`Provide
`Results
`455
`
`FIG. 5
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 6 of 17
`
`

`

`Patent Application Publication
`
`Oct. 8, 2015 Sheet 6 of 6
`
`US 2015/0286658 A1
`
`
`
`
`
`Receive
`Image
`
`Post image
`
`Receive
`Image
`
`Select 1
`Destination
`
`Receive
`Input
`
`Select 1
`Destination
`
`Receive 1
`Review
`
`Deliver 1
`Word
`
`Detect 2"
`Word
`
`Select 2"
`Destination
`
`Deliver2"
`Word
`
`Select 2"
`Destination
`
`Detect
`Completion
`
`Associate
`Tags
`
`FIG. 7
`
`Receive
`Tags
`
`Associate
`Tags
`
`FG.
`
`Receive 2"
`Review
`
`Deliver
`
`FIG. 8
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 7 of 17
`
`

`

`US 2015/0286658 A1
`
`Oct. 8, 2015
`
`IMAGE PROCESSING
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`0001. This application claims priority to and benefit of
`U.S. Provisional patent application entitled “Mobile Device
`Identification Application, filed May 1, 2013 and given Ser.
`No. 13/874,815 on filing (pending correction to a provisional
`application serial number); U.S. Provisional patent applica
`tion entitled “Visual Search filed Apr. 4, 2014 and having
`Ser. No. 61/975,691; U.S. Provisional patent application
`entitled “Visual Search Advertising.” filed Apr. 7, 2014 and
`having Ser. No. 61/976,494; and U.S. Provisional patent
`application entitled Image “Processing.” filed May 1, 2014
`and having Ser. No. 61/987,156. The above provisional patent
`applications are hereby incorporated herein by reference.
`
`BACKGROUND
`0002 1. Field of the Invention
`0003. The invention is in the field of image processing, and
`more particularly in the field of characterizing content of
`images.
`0004 2. Related Art
`0005. It is typically more difficult to extract information
`from images as compared to text data. However, a significant
`fraction of information is found in images. The reliability of
`automated image recognition systems is highly dependent on
`the contents of an image. For example, optical character
`recognition is more reliable than facial recognition. It is a goal
`of image recognition to tag an image. Tagging refers to the
`identification of tags (words) that characterize the content of
`an image. For example an image of a car may be tagged with
`the words “car,” “Ford Granada, or “White 1976 Ford
`Granada with broken headlight. These tags include varying
`amounts of information and, as such, may vary in usefulness.
`
`SUMMARY
`0006 Embodiments of the invention include a two
`pronged approach to tagging of images. The first prong is to
`perform automated image recognition on an image. The auto
`mated image recognition results in a review of the image. The
`image review includes one or more tags identifying contents
`of the image and optionally also a measure of confidence
`representative of the reliability of the automated image rec
`ognition. The second prong in the approach to tagging of
`images includes a manual tagging of the image. Manual tag
`ging includes a person viewing each image, considering the
`content of the image, and manually providing tags represen
`tative of the content of the image. Automated image recogni
`tion has an advantage in that the cost, in time or money, of
`analyzing each image can be relatively low. Manual tagging
`of images has an advantage of higher accuracy and reliability.
`0007 Embodiments of the invention combine both auto
`mated image recognition and manual image recognition. In
`Some embodiments automated image recognition is per
`formed first. The resulting image review typically includes
`both one or more tags characterizing the image and a measure
`of confidence in the accuracy of these tags. If the confidence
`is above a predetermined threshold, then these tags are asso
`ciated with the image and provided as an output of the tagging
`process. If the confidence is below the predetermined thresh
`old, then a manual review of the image is performed. The
`manual review results in additional and/or different tags that
`
`characterize the contents of the image. In some embodiments,
`the automated image recognition and the manual review of
`the image are performed in parallel. The manual review is
`optionally cancelled or aborted if the automated image rec
`ognition results in one or more tags having a confidence above
`the predetermined threshold.
`0008. In some embodiments recognition of an image can
`be upgraded. Upgrading of the image recognition process
`includes a request for further or improved tags representative
`of the content of the image. For example, if automated image
`recognition results in the tags “white car, an upgrade of this
`recognition may result in the tags “white Ford Granada. In
`Some embodiments, an upgraded review makes use of an
`expert human reviewer. For example, the above example may
`include the use of a human reviewer with an expert knowl
`edge of automobiles. Other examples of reviewer expertise
`are discussed elsewhere herein.
`0009 Various embodiments of the invention include fea
`tures directed toward improving the accuracy of image rec
`ognition while also minimizing cost. By way of example,
`these features include efficient use of human reviewers, real
`time delivery of image tags, and/or seamless upgrades of
`image recognition. The approaches to image recognition dis
`closed herein are optionally used to generate image tags Suit
`able for performing internet searches and/or selecting adver
`tisements. For example, in some embodiments, image tags
`are automatically used to perform a Google search and/or sell
`advertising based on Google's AdWords.
`10010 Various embodiments of the invention include an
`image processing system comprising an I/O configured to
`communicate an image and image tags Over a communication
`network; an automatic identification interface configured to
`communicate the image to an automatic identification system
`and to receive a computer generated review of the image from
`the automatic identification system, the computer generated
`review including one or more image tags identifying contents
`of the image; destination logic configured to determine a first
`destination to send the image to, for a first manual review of
`the image by a first human reviewer; image posting logic
`configured to post the image to the destination; review logic
`configured to receive the a manual review of the image from
`the destination and to receive the computer generated review,
`the manual review including one or more image tags identi
`fying contents of the image; response logic configured to
`provide the image tags of the computer generated review and
`the image tags of the manual review to the communication
`network; memory configured to store the image; and a micro
`processor configured to execute at least the destination logic.
`0011 Various embodiments of the invention include a
`method of processing an image, the method comprising
`receiving an image from an image source; distributing the
`image to an automated image identification system; receiving
`a computer generated review from the automated image iden
`tification system, the computer generated review including
`one or more image tags assigned to the image by the auto
`mated image identification system and a measure of confi
`dence, the measure of confidence being a measure of confi
`dence that the image tags assigned to the image correctly
`characterize contents of the image; placing the image in an
`image queue; determining a destination; posting the image
`for manual review to a first destination, the first destination
`including a display device of a human image reviewer, and
`receiving a manual image review of the image from the des
`tination, the image review including one or more image tags
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 8 of 17
`
`

`

`US 2015/0286658 A1
`
`Oct. 8, 2015
`
`assigned to the image by the human image reviewer, the one
`or more image tags characterizing contents of the image.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0012 FIG. 1 illustrates an image processing system,
`according to various embodiments of the invention.
`0013 FIG. 2 illustrates an image capture screen, accord
`ing to various embodiments of the invention.
`0014 FIG. 3 illustrates search results based on an image
`analysis, according to various embodiments of the invention.
`0015 FIG. 4 illustrates methods of processing an image,
`according to various embodiments of the invention.
`0016 FIG. 5 illustrates alternative methods of processing
`an image, according to various embodiments of the invention.
`0017 FIG. 6 illustrates methods of managing a reviewer
`pool, according to various embodiments of the invention.
`0018 FIG. 7 illustrates methods of receiving image tags in
`real-time, according to various embodiments of the invention.
`0019 FIG. 8 illustrates methods of upgrading an image
`review, according to various embodiments of the invention.
`
`DETAILED DESCRIPTION
`0020 FIG. 1 illustrates an Image Processing System 110,
`according to various embodiments of the invention. Image
`Processing System 110 is configured for tagging of images
`and may include one or more distributed computing devices.
`For example, Image Processing System 110 may include one
`or more servers located at geographically different places.
`Image Processing System 110 is configured to communicate
`via a Network 115. Network 115 can include a wide variety of
`communication networks, such as the internet and/or a cellu
`lar telephone system. Network 115 is typically configured to
`communicate data using standard protocols such as IP/TCP.
`FTP, etc. The images processed by Image Processing System
`110 are received from Image Sources 120 (individually
`labeled 120A, 120B, etc.). Image Sources 120 can include
`computing resources connected to the internet and/or per
`Sonal mobile computing devices. For example Image Source
`120A may be a web server configured to provide a social
`networking website or a photo sharing service. Image Source
`120B may be a Smartphone, camera, or other portable image
`capture device. Image sources may be identified by a univer
`sal resource locator, an internet protocol address, a MAC
`address, a cellular telephone identifier, and/or the like. In
`Some embodiments Image Processing System 110 is config
`ured to receive images from a large number of Image Sources
`120.
`Part of the image tagging performed by Image Pro
`0021
`cessing System 110 includes sending images to Destinations
`125 (individually labeled 125A, 125B, etc.). Destinations 125
`are computing devices of human image reviewers and are
`typically geographically remote from Image Processing Sys
`tem 110. Destinations 125 include at least a display and data
`entry devices such as a touch screen, keyboard and/or micro
`phone. Destinations 125 may include personal computers,
`computing tablets, Smartphones, etc. In some embodiments,
`Destinations 125 include a (computing) application specifi
`cally configured to facilitate review of images. This applica
`tion is optionally provided to Destinations 125 from Image
`Processing System 110. In some embodiments, Image Pro
`cessing System 110 is configured for human image reviewers
`to log into a user account from Destinations 125. Destinations
`125 are typically associated with an individual reviewer and
`
`may be identified by an internet protocol address, a MAC
`address, a login session identifier, cellular telephone identi
`fier, and/or the like. In some embodiments, Destinations 125
`include an audio to text converter. Image tagging data pro
`vided by a human image reviewer at a member of Destina
`tions 125 is sent to Image Processing System 110. The image
`tagging data can include textual image tags, audio data
`including verbalized tags, and/or non-tag information Such as
`upgrade requests or inappropriate (explicit) material desig
`nations.
`0022 Image Processing System 110 includes an I/O (in
`put/output) 130 configured for communicating with external
`systems. I/O 130 can include routers, switches, modems,
`firewalls, and/or the like. I/O 130 is configured to receive
`images from Image Sources 120, to send the images to Des
`tinations 125, to receive tagging data from Destinations 125,
`and optionally to send image tags to Image Sources 120.
`0023 Image Processing System 110 further includes
`Memory 135. Memory 135 includes hardware configured for
`the non-transient storage of data Such as images, image tags,
`computing instructions, and other data discussed herein.
`Memory 135 may include, for example, random access
`memory (RAM), hard drives, optical storage media, and/or
`the like. Memory 135 is configured to store specific data, as
`described herein, through the use of specific data structures,
`indexing, file structures, data access routines, security proto
`cols, and/or the like.
`0024 Image Processing System 110 further includes at
`least one Processor 140. Processor 140 is a hardware device
`Such as an electronic microprocessor. Processor 140 is con
`figured to perform specific functions through hardware, firm
`ware or loading of software instructions into registers of
`Processor 140. Image Processing System 110 optionally
`includes a plurality of Processor 140. Processor 140 is con
`figured to execute the various types of logic discussed herein.
`0025 Images received by Image Processing System 110
`are first stored in an Image Queue 145. Image Queue 145 is an
`ordered list of images pending review, stored in a sorted list.
`Images stored in Image Queue 145 are typically stored in
`association with image identifiers used to reference the
`images and may have different priorities. For example,
`images received from a photo sharing website may have
`lower priority than images received from a Smartphone. Gen
`erally, those images for which a requester is waiting to receive
`image tags representing an image in real-time are given
`higher priority relative to those for which the image tags are
`used for Some other purpose. Image Queue 145 is optionally
`stored in Memory 135.
`0026. Within Image Queue 145 images are optionally
`stored in association with an image identifier or index, and
`other data associated with each image. For example, an image
`may be associated with source data relating to one of Image
`Sources 120. The Source data can include geographic infor
`mation Such as global positioning system coordinates, a street
`and/or city name, a Zip code, and/or the like. The source data
`may include an internet protocoladdress, a universal resource
`locator, an account name, an identifier of a Smartphone, and/
`or the like. Source data can further include information about
`a language used on a member of Image Sources 120, a
`requested priority, a search request (e.g., an request to do an
`internet search based on image tags resulting from the image),
`and/or the like.
`0027. In some embodiments, an image within Image
`Queue 145 is stored in association with an indication of a
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 9 of 17
`
`

`

`US 2015/0286658 A1
`
`Oct. 8, 2015
`
`particular Subset of the image, the Subset typically including
`an item of particular interest. For example, a requestor of
`image tags may be interested in obtaining image tags relating
`to the contents of a particular Subset of an image. This can
`occur when an image includes several objects. To illustrate,
`considering an image of a hand with a ring on one of the
`fingers, the user may wish to designate the ring as being a
`particular area of interest. Some embodiments of the inven
`tion include an application configured for a user to specify the
`particular item of interest by clicking on the object or touch
`ing the object on a display of Image Source 120B. This
`specification typically occurs prior to sending the image to
`Image Processing System 110.
`0028. If an image is stored in association with an indica
`tion that a particular Subset of the image is of particular
`importance, then an Image Marking Logic 147 is optionally
`used to place a mark on the image. The mark being disposed
`to highlight the particular Subset. This mark may be made by
`modifying pixels of the image corresponding to the Subset
`and this mark allows a human image reviewer to focus on the
`marked Subset. For example, the image may be marked with
`a rectangle or circle prior to the image being posted to one or
`more of Destinations 125. In alternative embodiments, Image
`Marking Logic 147 is included within an application config
`ured to execute on one or more of Image Sources 120 or
`Destinations 125. Image Marking Logic 147 includes hard
`ware, firmware, and/or Software stored on a non-transient
`computer readable medium.
`0029. Under the control of Processor 140, images within
`Image Queue 145 are provided to an Automatic Identification
`Interface 150. The images are provided thus as a function of
`their priority and position in Image Queue 145. Automatic
`Identification interface 150 is configured to communicate the
`image, and optionally any data associated with the image, to
`an Automatic Identification System 152. Automatic Identifi
`cation Interface 150 is further configured to receive a com
`puter generated review of the image from Automatic Identi
`fication System 152, the computer generated review
`including one or more image tags identifying contents of the
`image. In some embodiments, Automatic Identification Inter
`face 150 is configured to communicate the image and data via
`Network 115 in a format appropriate for an application pro
`gramming interface (API) of Automatic Identification Sys
`tem 152. In some embodiments, Automatic Identification
`System 152 is included within Image Processing System 110
`and Automatic Identification Interface 150 includes, for
`example, a system call within an operating system or over a
`local area network.
`0030 Automatic Identification System 152 is a computer
`automated system configured to review images without a
`need for human input on a per picture basis. The output of
`Automatic Identification System 152 is a computer generated
`image review (e.g., a review produced without human input
`on a per picture basis.) Rudimentary examples of Such sys
`tems are known in the art. See, for example, Kooaba, Clarifai,
`Alchemy API and Catchoom. Automatic Identification Sys
`tem 152 is typically configured to automatically identify
`objects within a two dimensional image based on shapes,
`characters and/or patterns detected within the image. Auto
`matic Identification System 152 is optionally configured to
`perform optical character recognition and/or barcode inter
`pretation. In some embodiments, Automatic Identification
`System 152 is distinguished from systems of the prior art in
`that Automatic Identification System 152 is configured to
`
`provide a computer generated review that is based on the
`image Subset indication(s) and/or image source data, dis
`cussed elsewhere herein.
`0031. In addition to one or more image tag(s), a computer
`generated review generated by Automatic Identification Sys
`tem 152 optionally includes a measure of confidence repre
`sentative of a confidence that the one or more image tags
`correctly identify the contents of the image. For example, a
`computer generated review of an image that is primarily
`characters or easily recognizable shapes may have a greater
`confidence measure than a computer generated review of an
`image that consists of abstractor ill-defined shapes. Different
`automated image recognition systems may produce different
`confidence levels for different types of images. Automatic
`Identification Interface 150 and Automatic Identification Sys
`tem 152 are optional in embodiments in which automatic
`identification is performed by a third party.
`0032. Image Processing System 110 further includes a
`Reviewer Pool 155 and Reviewer Logic 157 configured to
`manage the Reviewer Pool 155. Reviewer Pool 155 includes
`a pool (e.g., group or set) of human image reviewers. Each of
`the human image reviewers is typically associated with a
`different member of Destinations 125. Memory 135 is option
`ally configured to store Reviewer Pool 155. In some embodi
`ments, the human image reviewers included in Reviewer Pool
`155 are classified as “active' and “inactive.” For the purposes
`of this disclosure, an active human image reviewer is consid
`ered to be one that is currently providing image tags or has
`indicated that they are prepared to provide image tags with
`minimal delay. In embodiments that include both active and
`inactive human image reviewers, the active reviewers are
`those that are provided image for review. The number of
`active reviewers may be moderated in real-time in response to
`a demand for image reviews. For example, the classification
`of a human image reviewer may be changed from inactive to
`active based on a number of unviewed images in Image Queue
`145. An inactive reviewer is one that is not yet active, that has
`let the review of an image expire, and/or has indicated that
`they are not available to review images. Inactive reviewers
`may request to become active reviewers. Inactive reviewers
`who have made such a request can be reclassified as active
`human image reviewers when additional active human image
`reviewers are needed. The determination of which inactive
`reviewers are reclassified as active reviewers is optionally
`dependent on a reviewer score (discussed elsewhere herein).
`0033 Reviewer Logic 157 is configured to manage
`Reviewer Pool155. This management optionally includes the
`classification of human image reviewers as active or inactive.
`For example, Reviewer Logic 157 may be configured to
`monitor a time that a human image reviewer takes to review
`an image and, if a predetermined maximum review time
`(referred to herein as an image expiration time), changing the
`classification of the human image reviewer from active to
`inactive. In another example, Reviewer Logic 157 may be
`configured to calculate a review score for a human image
`reviewer. In some embodiments, the review score is indicative
`of the completeness, speed and/or accuracy of image reviews
`performed by the particular human image reviewer. The
`review score can be calculated or changed based on review
`times and occasional test images. These test images may be,
`for example images placed in Image Queue 145 that have
`been previously reviewed by a different human image
`reviewer. The review score may also be a function of mon
`etary costs associated with the human image reviewer.
`
`ACCESSIBE LTD EXHIBIT 1007
`Page 10 of 17
`
`

`

`US 2015/0286658 A1
`
`Oct. 8, 2015
`
`Reviewer Logic 157 includes hardware, firmware, and/or
`Software stored on a non-transient computer readable
`medium. In some embodiments, reviewer scores are manu
`ally determined by human moderators. These human mod
`erators review images and the tags assigned to these images
`by human image reviewers. Moderators are optionally sent a
`statistical sampling of reviewed images and they assign a
`score to the tagging of the images. This score is optionally
`used in determining reviewer scores.
`0034. In some embodiments, Reviewer Logic 157 is con
`figured to monitor status of human image reviewers in real
`time. For example, Reviewer Logic 157 may be configured to
`monitor the entry of individual words or keystrokes as entered
`by a reviewer at Destination 125A. This monitoring can be
`used to determine which reviewers are actively reviewing
`images, which reviewers have just completed review of an
`image, and/or which reviewers have not been providing tag
`input for a number of seconds or minutes. The entry of tag
`words using an audio device may also be monitored by
`Reviewer Logic 157.
`0035. In some embodiments, members of Reviewer Pool
`155 are associated with a specialty in which the human image
`reviewer has expertise or special knowledge in. For example,
`a reviewer may be an expert in automobiles and be associated
`with that specialty. Other specialties may include art, plants,
`animals, electronics, music, food medical specialties, cloth
`ing, clothing accessories, collectables, etc. As is discussed
`elsewhere herein, a specialty of a reviewer may be used to
`select that reviewer during an initial manual review and/or
`during a review upgrade.
`0036. The review score and/or specialty associated with a
`human image reviewer are optionally used by Reviewer
`Logic 157 to determine which inactive reviewer to make
`active, when additional active reviewers are required.
`Reviewer Logic 157 includes hardware, firmware, and/or
`Software stored on a non-transient computer readable
`medium.
`0037 Image Processing System 110 further includes Des
`tination Logic 160. Destination Logic 160 is configured to
`determine one or more destinations (e.g., Destinations 125) to
`send an image to for manual review. Each of Destinations 125
`is associated with a respective human image reviewer of
`Reviewer Pool 155. The determinations made by Destination
`Logic 160 are optionally based on characteristics of the
`human image reviewer at the determined destination. The
`destination may be a computing device, Smartphone, tablet
`computer, personal computer, etc. of the human image
`reviewer. In some embodiments, the destination is a browser
`from which the reviewer has logged into Image Processing
`System 110. In some embodiments, determining the destina
`tion includes determining an MAC address, sessionidentifier,
`internet protocol and/or universal resource locator of one of
`Destinations 125. Destination Logic 160 includes hardware,
`firmware and/or software stored on a non-transient computer
`readable medium.
`0038. Typically, Destination Logic 160 is configured to
`determine Destinations 125 associated with active rather than
`inactive human image reviewers as determined by Reviewer
`Logic 157. Destination Logic 160 is also typically configured
`to determine Destinations 125 based on review scores of
`reviewers. For example, those reviewers having higher
`reviewer scores may be selected for higher priority reviews
`relative to reviewers having lower reviewer scores. Thus, the
`
`determination of a member of Destinations 125 can be based
`on both reviewer scores and image review priority.
`0039. In some embodiments, Destination Logic 160 is
`configured to determine one or more members of Destina
`tions 125 based on the real-time monitoring of the associated
`reviewers’ input activity. As discussed elsewhere herein, this
`monitoring may be performed by Reviewer Logic 157 and
`can include detection of individual words or keystrokes
`entered by a human image reviewer. In some embodiments,
`Destination Logic 160 is configured to favor selecting Desti
`nation 125A at which a human image reviewer has just com
`pleted a review of an image relative to Destination 125B at
`which a human image reviewer is currently typing image tags
`on a keyboard.
`0040. In some embodiments, Destination Logic 160 is
`configured to use image tags received via Automatic Identi
`fication System 152 to determine one or more members of
`Destinations 125. For example, if an image tag of “car is
`received via Automatic Identification Interface 150 then Des
`tination Logic 160 can use this information to select a mem
`ber of Destinations 125 associated with a human image
`reviewer that has a specialty in automobiles.
`0041. The value of an image review may also be consid
`ered in the selection of a destination for manual review. For
`example, an image review of high value may lead to the
`determination of a destination associated with a human image
`reviewer having a relatively high review score, while an
`image review of lower value may lead to the determination of
`a destination associated with a human image reviewer having
`a relatively lower review score. In some embodiments, for
`Some image reviews, Destination Logic 160 is configured to
`select among Destinations 125 so as to minimize a time
`required to review an image, e.g., to minimize a time until the
`image tags of the manual review are provided to Network 115.
`0042. Destination Logic 160 is optionally configured to
`determine multiple destinations for a single image. For
`example, a first destination may be selected and then, follow
`ing an upgrade request, a second destination may be deter
`mined. The upgrade request may come from the Image
`Source 120A or from a human image reviewer associated
`with the