as) United States
`a2) Patent Application Publication co) Pub. No.: US 2010/0142759 Al
` Ridleyetal. (43) Pub. Date: Jun. 10, 2010
`
`
`
`US 20100142759A1
`
`(54) ASYSTEM AND A METHOD FOR
`DETECTING A DAMAGED OR MISSING
`MACHINE PART
`
`(75)
`
`Inventors:
`
`Rodney K.Ridley, Edmonton
`(CA); Tadeusz Kazmierczak,
`Edmonton (CA); Linsong Cai.
`Edmonton (CA); Paul Charles
`Johnston, Edmonton (CA); Helen
`:
`,
`:
`s
`Pinto, Calgary (CA); Jun Sun,
`Calgary (CA); Mare Provencher,
`Calgary (CA); John Harry
`Woolley, Edmonton (CA); Garry
`Cardinal, Edmonton (CA)
`
`Correspondence Address:
`RODMAN RODMAN
`10 STEWART PLACE, SUITE 2CE
`WHITE PLAINS, NY10603 (US)
`
`(73) Assignee:
`
`ALBERTA RESEARCH
`COUNCIL INC., Edmonton, AB
`(CA)
`
`(21) Appl. No.:
`
`12/300,257
`
`(22)
`
`PCT Filed:
`
`May 4, 2007
`
`(86) PCT No.:
`
`PCT/CA07/00776
`
`§ 371 (0),
`(2), (4) Date:
`Nov. 10, 2008
`Foreign Application Priority Data
`
`(30)
`
`(CA) Sh, gatas quer eae eReet 2,546,758
`May 12, 2006
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`G06K 9/00
`(52) US. Ch. occcececesses cesersesnsesneeeaes 382/107
`(57)
`ABSTRACT
`
`A system and a methodfordetecting a damaged or missing
`machinepart. The system includes an image capturing device
`for capturing images of the machine and a processor for
`processing the captured images. The system may further
`include a sensible output for providing an indication of a
`damaged or missing machine part. The method includescap-
`turing images of the machine against a background which
`movesrelative to the machine overtime, selecting a pair of
`time-separated images from the captured images, generating
`a displacement imagefrom the pair of images, comparing the
`machinefrom the displacement image with a machine model,
`and identifying a damaged or missing machine part from the
`comparison of the displacement image with the machine
`model. The method mayfurther include providing a sensible
`output which indicates a damaged or missing machinepart.
`
`“11 Power Cable
`Zz Camera Cable
`3) Touch Screen DisplayCable
`“4! camp Cable: CustomerSupplied
`5 Amphenol PTOGE-12-35(SR)
`_&) Amphenol PTO2E-12-3"
`<1.) WaterTight Conduit
`8)Customer Tesninates Lamp Cables
`3 Amprenct PTOSE- 14-48S15R}
`19) Amphenol PTO2E. 14-16P
`4 Amprenol PTO2E-14-188
`2 Amphenol PTOBE-14-74F{SR]
`{31 Amphenol PTOZE-14-95
`4} Amphenol PTODE-14-*9F(SR]
`16) Amphend PTOGE-12-2P(5R}
`47} Molex CONMOLFHT2PF
`18} Molex 02-06-2122
`
`
`
`
`
`
` 1Shamphend PTOZE4238
`
`
`
`
`Supply Enclosure
`
`yo | 34
`
`Equipment Locatedin
`Shovet Electrical Room
`
`Equipment Located in
`Shovel Cabin
`
`Equipment Located on
`Shovel Boom
`
`20
`
`Shenzhen Tuozhu 1014
`
`1
`
`Shenzhen Tuozhu 1014
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 1 of 6
`
`US 2010/0142759 Al
`
`
`
`wangtone(Esaweysng(MSISE-ZL-AgQLetusydury(SpanddngceigeoduetbaiqepKeydsig
`
`
`
`ajgepesauregZz
`
`araJerod(+
`
`BHD
`
`
`
`
`
`
`d€-71-320Le1ouaudury(3°
`
`
`
`
`
`saige9due]sajaumwe,won(7
`
`lasisat-rt-a901dPusydury16
`1uayduny«JRL-pL-azOLdioueudunyor
`S@t-P1-3Z01.4
`
`Qughdat-bi-god.glouauduryZP
`SB)-Pt-BZO0LdusudunyET
`
`
`
`
`unpuesBysare(2°
`
`
`
`
`
`(USId6t-PL-390LdHusyduTy(FT
`
`
`
`
`
`ZZL2-90-20XBWIN(BE
`
`
`
`GasldeZi-adnigpuaydusyGi
`
`SETHAOLpusyduryGi
`
`ddZbHIOWNODwowZL
`
`
`
`udpayeso07wWeawdinbs
`
`
`uypayeue7puaudinby
`
`
`
`u)payaao7>juatudinbS
`
`ainsapuyAiddng
`
`Of
`
`
`
`Wwoog}aAc"US
`
`
`
`
`
`wooyjeIUupalywACUS
`
`2
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 2 of 6
`
`US 2010/0142759 Al
`
`MIDS by Sensors Engineering, Alberta Research Council
`
`
`
`
`
`
`
`RT.
`
`RESEARCH 3 XBBBX,XXXXK,OOK,KKK
`
`cou
`
`NCIL
`
`gage) ad a
`
`ACKNOWLEDGE
`
`SETUP
`
`CANCEL
`
`2006-04-11
`13:27:57
`
`Figure 2
`
`24
`
`3
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 3 of 6
`
`US 2010/0142759 Al
`
`nes by Sensors Engineering, Alberta Research Council
`
`
`
`
`
`
`BERTA
`iene 3
`COUNCIL
`
`dadaaaa
`SEBEN (Sz
`vo
`sess cuentas
`
`2006-04-11
`13:28:51
`
`Figure 3
`
`24
`
`4
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 4 of 6
`
`US 2010/0142759 Al
`
`
`
`
`Figure 4
`
`Figure 5
`
`5
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 5 of 6
`
`US 2010/0142759 Al
`
`
`
`Figure 6
`
`
`
`Figure 7
`
`
`
`Figure 8
`
`6
`
`

`

`Patent Application Publication
`
`Jun. 10,2010 Sheet 6 of 6
`
`US 2010/0142759 Al
`
`
`54
`
`Figure 10
`
`Figure 12
`
`7
`
`

`

`US 2010/0142759 Al
`
`Jun. 10, 2010
`
`A SYSTEM AND A METHOD FOR
`DETECTING A DAMAGED OR MISSING
`MACHINE PART
`
`TECHNICAL FIELD
`
`[0001] A system and a methodfor detecting a damaged or
`missing machine part using image analysis and computer
`vision techniques.
`
`BACKGROUNDOF THE INVENTION
`
`
`
`technologies have been proposed for
`[0002] Various
`inspecting machines and/or detecting and reporting inci-
`dences of damage to machine parts.
`[0003]
`Japan Patent Application No. 07257561 (Fumiaki et
`al) describes a method for detecting a wrong or missing
`engine part which involves capturing images using a CCD
`camera and comparing detection data with reference data
`based upon the brightness distribution within regionsof the
`captured images.
`[0004] U.S. Pat. No. 4,399,554 (Perkins, III et al) describes
`a system for inspecting engine head valve spring assemblies
`for missing retainer keys which comprisesa solid state cam-
`era for taking picturesofa valve spring assembly andprovid-
`ing picture data, a position encoder whichrelates the position
`of the engine head to the camera, and a computer which
`analyzes the picture data to determine the center ofa digitized
`imageofa valve spring assembly and anintensityprofile of
`the expectedlocation ofthe retainer keysrelative to the center
`ofthe valve spring assembly, whichintensity profile is used to
`determine whethera retainer keyis missing.
`[0005] U.S. Pat. No. 5,743,031 (Launderet al) describes an
`apparatusfor providing a signal indicative of loss or immi-
`nent loss of digging hardware. The apparatus includes an
`actuatable indicator and an actuator. In the preferred embodi-
`ments the actuator is comprisedof a lanyard whichis secured
`between an adaptor and a digging tooth.If the digging tooth
`breaks off or becomes dislodged from the adaptor, the lanyard
`senses the change in predeterminedrelationship between the
`adaptor and the digging tooth and actuates the actuatable
`indicator. In the preferred embodimentsthe actuatable indi-
`cator is comprised of a smoke canister.
`[0006] U.S. Pat. No. 6,796,709 (Choi) describes a method
`and system for estimating turbine bucket oxidation condition.
`The method includes measuring with an infrared camera a
`emperature distribution on a surface of a rotating turbine
`bucket, determining a condition index based upon the tem-
`perature distribution, and estimating the turbine bucketoxi-
`dation condition based on the condition index. The system
`includes the infrared camera, a triggering mechanism
`coupled with the infrared camerafortriggering the cameraat
`predetermined intervals based upontherotating speed of the
`turbine, and a processor for receiving the output from the
`infrared camera and for determining the condition index.
`[0007] U.S. Pat. No. 6,870,485 (Lujan et al) describes a
`nethod and apparatusfor detecting andreporting dislocation
`of heavy metal parts on mining equipment. The apparatus
`includes a spring loaded switch sandwiched between heavy
`netal parts, which upon partial separation of the parts
`expands and turns on anelectrical switch to activate a radio
`ransmitter, sending an alarm signalto a receiverat a remote
`location.
`
`
`
`[0008] United States Patent Application Publication No.
`US 2005/0081410 Al (Furem etal) describes a system and
`
`method for distributed reporting of machine performance.
`The system is comprised of a machine data management
`system which permits information relating to a machine to be
`gathered and analyzed while the machine 1s operating.
`[0009] Another technology proposed by the University of
`Alberta (Xiujuan Luo) uses laser range data for detecting
`missing shovel teeth. The technology involves creating a
`CAD modelof an intact tooth, using a laser rangefinder to
`scan the tooth line of a shovel, and comparing the laser scan
`with the CAD model to detect missing teeth.
`[0010]
`Finally, Motion Metrics International Corp. ofVan-
`couver, British Columbiaoffers a broken tooth detection sys-
`tem for mining shovels and loaders under the trade-mark
`ToothMetrics™.
`
`SUMMARYOF THE INVENTION
`
`[0011] The present invention is a system and a method for
`detecting a damaged or missing machinepart. The invention
`mayalso be comprised ofa computer which is programmedto
`perform aspects of the method. The invention mayalso be
`comprised of a computer readable medium which contains
`computer readableinstructionsfor performing aspects of the
`method. The invention may also be comprised of a signal
`whichis operable to cause a processor to perform aspects of
`the method.
`
`[0012] The machine maybe anytype of machine and the
`machinepart maybe comprised of anytype ofpart associated
`with the machine.
`[0013]
`Ina first broad system aspect, the invention is com-
`prised of:
`(a) an image capturing device such as a camera,
`[0014]
`for capturing images of a machine against a background
`which movesrelative to the machine over time; and
`[0015]
`(b) a processor for processing the captured
`images to determine whether one or more machineparts
`is damaged or missing.
`the
`[0016]
`In preferred embodiments of the invention,
`machine is a mining shovel and the machinepart is a tooth on
`the bucket of the shovel.
`
`Ina second system aspect, the mvention is com-
`[0017]
`prised of:
`(a) an image capturing device such as a camera,
`[0018]
`for capturing images of a bucket tooth line against a
`background which movesrelative to the bucket over
`time; and
`[0019]
`(b) a processor for processing the captured
`images to determine whether one or more teeth on the
`bucket is damaged or missing.
`[0020] The processoris preferably programmed to perform
`aspects of the methodof the invention.
`[0021] The system of the invention mayalso be comprised
`ofa sensible output for providing an indication of a damaged
`or missing tooth. The sensible output may be comprised ofa
`visual display, an audible alarm, or any other type of output
`which maybe sensed bya person orapparatus. For example,
`the sensible output may be comprised of a signal which
`causes the mining shovel to stop operating when a damaged
`or missing tooth is detected.
`[0022] The image capturing deviceis preferably mounted
`on the mining shovel so that a clear image ofthe bucket tooth
`line maybe obtained at some point during the operation ofthe
`mining shovel as a load is taken up by the mining shovel,
`moved to an unloading position and then dumped at the
`unloading position. Preferably the image capturing deviceis
`
`8
`
`

`

`US 2010/0142759 Al
`
`Jun. 10, 2010
`
`mounted on the mining shovel so that a clear image of the
`bucket tooth line may be obtained immediatelyafter the load
`is dumped at the unloadingposition.
`[0023] The processor maybe located at any suitable posi-
`tion on or within the mining shovel. The processor mayalter-
`natively be located remotely ofthe mining shovel. A commu-
`nication link is provided between the image capturing device
`and the processor so that the captured images may be pro-
`vided to the processor for processing. The communication
`link may be comprised of any suitable type oflink, including
`wired communication links and wireless communication
`links.
`Ina first broad method aspect, the invention is com-
`[0024]
`prised of:
`(a) capturing images ofa machine against a back-
`[0025]
`ground which movesrelative to the machine overtime;
`[0026]
`(b) selecting two time-separated images which
`reflect movement of the background relative to the
`machine;
`[0027]
`(c) generating a displacement image from the two
`time-separated images, wherein the displacement image
`provides an indication of movement of each pixel rep-
`resented in the displacement image between the two
`time-separated images;
`[0028]
`(d) comparing the machine from the displace-
`ment image with a model machine; and
`[0029]
`(e) identifying a damaged or missing machine
`part from the comparison of the machine from the dis-
`placement image and the model machine.
`[0030] The second method aspect may be further com-
`prised ofproviding a sensible output whichindicates a dam-
`aged or missing machinepart.
`[0031]
`In preferred embodiments the method of the inven-
`tion involvesthe capturing ofat least two imagesof a mining
`shovel bucket with an image capturing device such as acam-
`era, while the mining shovelis operating. For example, if two
`images are captured at different times while the mining
`shovel is operating, the apparent movementofpixels repre-
`senting the bucket and the apparent movementof pixels rep-
`resenting the background of the image will be different, as
`long as the bucket moves relative to the background during
`the time between the capturingofthe two images. Thediffer-
`ence in movement can be used to determine which pixels in
`the imagesrepresentthe bucket and whichpixels in the image
`represent the background. Moreparticularly, a displacement
`image may be generated from the two images, which dis-
`placement image will provide an indication of which pixels
`have “moved” between the two images and which images
`have not moved between the two images.
`[0032]
`In a second method aspect, the invention is com-
`prised of:
`(a) capturing images of the bucket tooth line
`[0033]
`against a background which movesrelative to the bucket
`over time;
`[0034]
`(b) selecting two time-separated images which
`reflect movement of the background relative to the
`bucket;
`[0035]
`(c) generating a displacement image from the two
`time-separated images, wherein the displacement image
`provides an indication of movement of each pixel rep-
`resented in the displacement image between the two
`time-separated images;
`[0036]
`(d) comparing the bucket tooth line fromthe dis-
`placement image with a model bucket tooth line; and
`
`(e) identifying a damagedor missing tooth from
`[0037]
`the comparison of the bucket tooth line from the dis-
`placement image and the model buckettooth line.
`[0038] The second method aspect may be further com-
`prised of providing a sensible output whichindicates a dam-
`aged or missing tooth.
`[0039]
`Ina third method aspect, the invention is comprised
`of:
`
`(a) capturing a numberof time-separated images
`[0040]
`ofa bucket tooth line against a background which moves
`relative to the bucket over time;
`[0041]
`(b) initial processing one or more pairs of the
`captured imagesto select a suitable pair of images from
`the gathered images, having regard to one or more ofthe
`following:
`tooth line can be
`[0042]
`(@) whether the bucket
`adequately identified in the pair of images(the bucket
`is not completely fixed relative to the image capturing
`device, but may raise and lowerrelative to the image
`capturing device and may move toward or away from
`the image capturing device. As a result, the bucket
`tooth line is adequatelyidentified in the pair ofimages
`whenit is within a defined position zoneofthe image
`andhas a size in the image whichis within a defined
`size range);
`[0043]
`(41) whether the backgroundis moving relative
`to the image capturing device and the bucket tooth
`line (since the method of the invention requires the
`generation of displacement images, the method will
`not work if there is no movement ofthe background
`relative to the image capturing device and the bucket
`tooth line); and
`[0044]
`(iti) whether the magnitude of the relative
`movementof the background between two images in
`a pair of images is within a desired range (too little
`movement reduces the resolution of the method and
`too much movementtaxes the processing capabilities
`ofthe system by requiring a larger area of the images
`to be processed);
`[0045]
`(c) final processing the suitable pair of images to
`generate a displacement imagefrom the suitable pair of
`images, wherein the displacement image provides an
`indication of movementof each pixel represented in the
`displacement image between the two imagesin the suit-
`able pair of images;
`[0046]
`(d) comparing the bucket tooth line from the dis-
`placement image with a model buckettooth line; and
`[0047]
`(e) identifying a damaged or missing tooth from
`the comparison of the bucket tooth line from the dis-
`placement image and the model buckettooth line.
`[0048] The third method aspect maybe further comprised
`of providing a sensible output which indicates a damaged or
`missing tooth.
`[0049]
`In a fourth method aspect, the invention is com-
`prised of:
`(a) capturing a sequence of images using an
`[0050]
`image capturing device such as a video camera;
`[0051]
`(b) initial processing one or more pairs of the
`captured imagesto select a suitable pair of images from
`the gathered images;
`[0052]
`(c) generating a displacement image from the
`suitable pair ofmages, whereinthe displacement image
`provides an indication of movement of each pixel rep-
`
`9
`
`

`

`US 2010/0142759 Al
`
`Jun. 10, 2010
`
`
`
`resented in the displacement image between the two
`images in the suitable pair of images;
`[0053]
`(d) locating a first boundary of the bucket in the
`displacement image;
`[0054]
`(e) locating a second boundary of the bucket in
`the displacement image;
`[0055]
`(f) creating a model bucket tooth line using the
`coordinatesofthe first boundary and the second bound-
`ary;
`(g) comparing the bucket tooth line fromthe dis-
`[0056]
`placement image with the model buckettooth line; and
`[0057]
`(h) identifying a damaged or missing tooth from
`the comparison of the bucket tooth line from the dis-
`placement image and the model bucket toothline.
`[0058] The fourth method aspect may be further comprised
`of providing a sensible output whichindicates a damaged or
`missing tooth.
`[0059] The processing of images, includingtheinitial pro-
`cessing,the final processing, and comparing the bucket tooth
`line from the displacement image with the model bucket tooth
`line, may be performed using any suitable computervision,
`image motion,optical flow, image matchingor pattern match-
`ing method. For example, the processing of pairs of images
`maybe performed using techniques used in stereo matching.
`The processing of images is preferably performed using a
`computer which is directed by computer codes to perform
`aspects of the method.
`[0060]
`Inpreferred embodiments, the processing ofimages
`is performedusing pattern matchingor block matching meth-
`ods. Such pattern matching or block matching methods may
`include various matching criteria and/orsearchstrategies. As
`non-limiting examples, matching criteria may include maxi-
`mum cross-correlation, minimum mean square error, mini-
`mum meanabsolute difference and maximum matchingpixel
`count methods. As non-limiting examples, search strategies
`mayinclude three step search or cross search methods.
`[0061]
`Pattern matching or block matching methods which
`are suitable for use in someorall aspects ofthe method ofthe
`invention may be embodied in commercially available soft-
`ware or may be performed using custom applications.
`[0062] Asan example of commercially available software,
`Matrox Imaging, a division of Matrox Electronic Systems
`Ltd. of Dorval, Quebec has developed the Matrox Imaging
`Library (MIL) as a developmentkit for pattern matching and
`block matching software solutions. One or more modules
`within the MIL maybe usedforaspects of the method of the
`invention.
`
`[0063] The generation of displacement images from cap-
`tured images maysimilarly be performed using commercially
`available software, or may be performed using custom appli-
`cation for the generation of the displacement images.
`[0064]
`In the preferred embodiments of the invention, the
`processing of the images, including pattern matching/block
`matching and the generation of displacement images from
`pairs of images, is performed using a combination of com-
`mercially available software and custom applications which
`both apply methods knownin theart of image processing.
`[0065] The initial processing of pairs of images may be
`performed in a manner which minimizesthe initial process-
`ing time. Theinitial processing time may be minimized using
`techniques knownin theart, by specifying processing param-
`eters and by defining search constraints. However, theinitial
`processing of images is preferably performed using a rela-
`tively large area of the images, since oneofthe goals of the
`
`
`
`initial processing is to determinethe magnitudeoftherelative
`movementofpixelsin the pair of imagesin orderto assess the
`suitability of the pair of imagesforfinal processing.
`[0066]
`Thefinal processing ofpairs of imagesis preferably
`performed in a manner which balancesthe required accuracy
`of the method with the required speed of the method. As a
`result of the initial processing, the final processing in many
`applications may be performedusing a relatively small area
`ofthe images, since the expected position ofthe pixels in the
`two imageswill to some extent be known.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`Tr
`
`
`
`invention will now be
`[0067] Embodiments of the
`described with reference to the accompanying drawings,in
`which:
`
`FIG. 11s a block diagram of a system for detecting
`[0068]
`a damagedor missing tooth on a mining shovel, according to
`a preferred embodiment ofthe invention.
`[0069]
`FIG. 2 isa view ofa touch screen monitor according
`toa preferred embodimentofthe invention uponthedetection
`of a damaged or missing bucket tooth.
`Tr
`[0070]
`G.3 is a view of a touch screen monitor according
`to a preferred embodiment of the invention following the
`touching of
`the “Acknowledge”button in the viewof FIG.2.
`TI
`[0071]
`FIG. 4 is a first captured image in a sequence of
`captured images.
`[0072]
`FIG. 51s a second captured image in a sequence of
`captured images.
`TI
`[0073]
`FIG. 6 is a displacement image generated from the
`captured images from FIG. 1 and FIG.2.
`[0074]
`FIG. 7 is amodel ofan upperleft corner ofa bucket,
`representing a first boundaryof the bucket.
`[0075]
`FIG. 8 is a schematic drawing depicting the dis-
`placement image of FIG. 3 with the model of FIG. 4 super-
`imposed thereon.
`[0076]
`FIG. 9 is a model of an upper right corner of a
`bucket, representing a second boundary ofthe bucket.
`[0077]
`FIG. 10 is a schematic drawing depicting the dis-
`placement image of FIG. 3 with the model of FIG. 6 super-
`imposed thereon.
`[0078]
`FIG. 11 is a model bucket tooth line created using
`the coordinates ofthe first boundary of the bucket and the
`second boundary of the bucket.
`TE
`[0079]
`FIG. 12 is a schematic drawing depicting the dis-
`placement image of FIG. 3 with the model bucket tooth line
`superimposedthereon.
`
`DETAILED DESCRIPTION
`
`[0080] Determining the presence or absence ofa tooth on a
`mining shovel by inspection of a single image taken bya
`camerais difficult. Iftwo time-separated imagesofthe bucket
`of the mining shovel are captured while the mining shovelis
`being operated, the apparent movement ofbackground pixels
`and bucket pixels (i.e., foreground) can be different. This
`difference in movement can be used to determine which pix-
`els in the captured imagesare part ofthe bucket (foreground)
`and which pixels are part of the background.
`[0081] The present invention is a remote ‘machinevision’
`technology, which in the preferred embodiments utilizes
`video camera output and specialized computeralgorithmsto
`monitor bucket teeth on a mining shovel.
`[0082] The buckettooth line of the bucket is analyzed on
`each upswing of the mining shovel and compared against a
`
`10
`
`10
`
`

`

`US 2010/0142759 Al
`
`Jun. 10, 2010
`
`[0083] With the system of the invention, shovel operators
`are alerted to partial or complete tooth breakageas soonasthe
`shovel comesinto the viewing range of the camera.
`[0084] The inventionoffers the following benefits to amine
`site:
`
`(a) prevents broken bucket teeth from damaging
`[0085]
`the crusher, conveyer belts, screens, pumps and other
`expensive equipment;
`[0086]
`(b) minimizes downtime by detecting broken
`teeth at the time of the breakage incident; and
`[0087]
`(c) improves overall efficiency in the mining
`operation through constant monitoring of the bucket
`tooth line status in termsof teeth wear.
`
`base-case scenario ofafullyintact tooth line. Whenatooth is [0102] The powersystems are designed in accordance with
`
`partially or completely broken or missing, the system auto-
`the user’s specifications. In the preferred embodiments, 120
`matically alerts the shovel operator by a sensible output in the
`Volt AC is provided to the system (20). Poweris converted to
`form of a visual alarm on a touch screen monitor.
`12 or 24 Volt DC for the lamps (32), the monitor (24) and the
`CPU(26). The lamp (32) poweris supplied directly from the
`power supply enclosure (34). Power for the camera (22) and
`the monitor (24) is supplied through the processor enclosure
`(30) via cables that supply electrical power and transmit
`image capture (camera) and touch screen controls (touch
`screen monitor).
`[0103] An optional data storage device (36) (preferably a
`+200 Gigabyte portable hard drive) maybe included. This
`extended data logging capability facilitates troubleshooting
`and resolution ofany site-specific issues that may arise in the
`course ofinstalling and using the system. Wherea data stor-
`age device (36) is provided, a separate data storage device
`enclosure (38) may be provided for this component.
`[0104]
`Theuserinterface is preferably an industrial touch
`screen monitor (24).
`[0105] The processor (26), power supply elements and
`optional data storage device (36) are preferably placed in
`separate NEMA IV enclosures that may be placed in an
`electrical room on the mining shovel (not shown).
`[0106] The processor enclosure (30) contains the process-
`ing hardware (26) and the power supply(40) for the camera
`(22) and touch screen monitor (24). From this enclosure, the
`camera (22) and monitor (24) are powered. There are three
`Mil-Spec connectors on the bottom of the processor enclo-
`sure (30) for DC Power in, a cable for the touch screen
`monitor(24), and the cables for the camera (22). A data port
`(42) is also located on the processor enclosure (30), which
`data port (42) includes a USB connectorand a network con-
`nector. The USB connector can be used by the data storage
`device (36) or by a USB memory stick, for downloading data.
`The network connection can be used bya laptop computerfor
`diagnostic purposes.
`[0107] The operating system ofthe processor(26) utilizes a
`Windows-XP embedded system which has been designed to
`reliably handle power interruptions without corrupting or
`hangingup the processor(26). The processor (26) boots when
`poweris present andflicking power off/on at any time during
`shovel operations1s acceptable.
`[0108] The processing software is stored on a computer
`readable medium such as a compactflash card (28) which
`inserts into the processing hardware (26). Details ofthe meth-
`ods and algorithms which are included in the processing
`software are provided below in connection with the descrip-
`tion of a preferred embodimentof the method ofthe inven-
`tion.
`
`Inthe preferred embodiment, the system (20)ofthe
`[0088]
`invention comprises a number of components that make up
`the hardware and provide the software which includes the
`algorithmsfor system operation.
`[0089] The features of the preferred embodiment of the
`system (20) can be summarized asfollows:
`[0090]
`(a)
`industrial-grade components, no moving
`parts, rugged construction;
`[0091]
`(b) a lightweight, vibration resistant video cam-
`era (22) as an image capturing device, designed for
`rugged outdoor operation, combined with a sun shroud
`for protection;
`[0092]
`(c) a compact (10.4") rugged LCD monitor (24)
`with resistive touch screen display. When a missing
`tooth is detected, the operator can both check the sen-
`sible output (1.e., visual alarm) and inspect a real time
`image ofthe buckettooth line;
`[0093]
`(d) fully molded connectors designed for rugged
`application, combined with cable wiringthatis resistant
`to severe weather conditions and rugged shovel opera-
`tions;
`(e) PC 104 computing hardware (26) with a Win-
`[0094]
`dows XP embedded operating system;
`[0095]
`(f) software providing image matching/pattern
`matching algorithms, which software resides on a com-
`puter readable medium such as a compactflash card (28)
`which inserts into the processor(26);
`[0096]
`(g) NEMAIVenclosures to protect the compo-
`nents of the system (20).
`[0097] A block diagram of the components of the system
`(20), according to a preferred embodimentofthe invention,is
`provided as FIG. 1.
`the following provides a
`[0098] Referring to FIG. 1,
`description of the componentsof the system (20).
`[0099] The camera (22) is provided by Kongsberg from
`Scotland. The required field of view for each shovel applica-
`tion is determined andthe required field of view is provided
`for in the camera (22).
`[0100] The central processing unit (CPU) (26) processor
`
`enclosure (30) is designed in accordance with NEMA IV
`specifications and incorporates Mil-Spec connectors. This
`processorenclosure (30), which is sealed and weatherproof,
`shouldbe placed in the shovel instrument room (not shown).
`[0101] The high intensity discharge (HID) lamps (32) are
`model number Hella AS200FF Xenon HID lamps, manufac-
`tured by Hella KGaA Hueck & Co.
`
`
`
`[0109] The power supply enclosure (34) contains the ele-
`ments that connect to the 120 Volt AC input provided on the
`mining shovel. 120 Volt AC is run into the power supply
`enclosure (34) and connected to the terminals. The connec-
`tion must be a sealed connection to ensure conformance to
`NEMA IVspecifications. In the preferred embodiment the
`120 Volt AC poweris converted and provided to the power
`inlet of the processor enclosure (30) as 24V DC.
`[0110]
`Powerfor the lamps (32) 1s provided from the power
`supply enclosure (34) directly to the lamps (32) as 24 Volt DC.
`A feature provided in the power supply enclosure (34) 1s a
`time-delayrelay which turnson lamps sequentiallyin orderto
`avoid excessive power draw onstart-up (the lamps require 10
`Ampseachto start, but normal current usage is 1.6 Amps for
`each lamp).
`
`11
`
`11
`
`

`

`US 2010/0142759 Al
`
`Jun. 10, 2010
`
`[0111] The data storage device enclosure (38) contains the
`data storage device (36) and is preferably installed in the
`electrical room of the mining shovel.
`[0112] The image capturing device (22) is preferably a high
`resolution monochrome video camera Model 0414-6002-
`
`
`
`
`
`
`
`002, manufactured by Kongsberg.
`[0113] The lamps (32) are 35 watts and require 24 Volt DC
`power. Preferablytwo (2) lamps (32) are used to illuminate
`he bucket. The lamps (32) are resistant to mechanical vibra-
`ion and shock, but care must be exercised when the lamps
`32) are in use or being transported.
`[0114] The camera (22) and the lamps(32) are preferably
`placed separately on the shovel boom (not shown).
`[0115] The camera mounting bracket preferably can be
`loosenedto allowaltering the camera pan position. Preferably
`both the camera (22) and lamps (32) are capable of both
`panning andtilting movement.
`[0116] The lamp power cableor cables are preferably Tech
`Cable #14, armoured PVC cable. Conduit Tee LB junction
`nits are preferably provided for each lamp (32) to allow
`connection of the lamp powercable to the two lamps (32).
`This conduit Tee is weatherproofand is designed for rugged
`applications and is preferably placed on the lamp mount.
`[0117] Acompact (10.4"), rugged,flat panel LCD monitor
`(24), with resistive touch screen, is preferably provided as the
`operatorinterface. This monitor (24)is preferablyplaced in
`the cab ofthe mining shovel within reach ofthe shovel opera-
`tor. The monitor(24) is connected to the processor enclosure
`(30) via a single cable. This cable provides the power, VGA
`signal, and touch screen communications. The screen ofthe
`monitor (24) provides an image ofthe bucket and a visible
`alarm in the eventof detection of damaged, broken or missing
`teeth. The interface is designed to show initially a yellow dot
`on a specific tooth location where there might be damage or
`significant wear. A red dot will appearif a full tooth missing.
`[0118] The preferred monitor (24) is a model LMV10 pro-
`vided by Datalux. Details of the specifications of the pre-
`ferred monitor (24) can be found at www.datalux.com.
`[0119] Mil-Spec cable connectors using adhesive heat
`shrink are preferably provided for all cables in orderto pro-
`vide weather protection.
`[0120] The camera cablesare preferably provided by Intec
`Video Systems. The main function of the cable jackets is to
`protect the primaryinsulation from environmental damage.
`The Intec Video Systems cables have polyurethane cable
`jackets that offer high performance and durability by provid-
`ing long-lasting protection in applicationsrequiring low-tem-
`perature flexibility, good weathering properties and resis-
`tance to wet environments.
`
`Polyurethane consistently outperforms conven-
`[0121]
`tional rubber compounds; its abrasion resistance makespoly-
`urethane superior to copolyester and thermoplastic polyole-
`fins, and it also offers superior protection from physical
`damage. Polyur