United States Patent (19)
`Birk et al.
`
`11)
`45
`
`4,146,924
`Mar. 27, 1979
`
`(73) Assignee:
`
`(54) SYSTEM FOR VISUALLY DETERMINING
`POSITION IN SPACE AND/OR
`ORIENTATION N SPACE AND APPARATUS
`EMPLOYNG SAME
`75) Inventors:
`John R. Birk, Peacedale; Robert B.
`Kelley, Kingston, both of R.I.; David
`A. Seres, Newark, Del.
`Board of Regents for Education of the
`State of Rhode Island, Providence,
`R.I.
`(21) Appl. No.: 615,716
`(22
`Filed:
`Sep. 22, 1975
`51) Int. C.’....................... G05B 19/42; G06F 15/46
`52 U.S.C. ........................................ 364/513; 414/5;
`318/568; 318/640; 358/903; 364/515; 364/559
`(58) Field of Search ................ 235/151, 151.1, 151.11;
`444/1; 340/172.5; 364/513, 514, 120, 559, 300;
`214/1 CM; 178/DIG. 21, DIG. 22, DIG. 36;
`318/640, 567,568
`
`References Cited
`(56)
`U.S. PATENT DOCUMENTS
`V
`3,216,311 11/1965
`Biberro et al. .............. 178/DIG. 36
`
`7/1969 Bridges ............................ 214/1 CM
`3,454,169
`6/1971 Hackmann et al. .
`214/1 CMX
`3,589,134
`6/1972 Besson et al. .................... 318/640X
`3,669,549
`3,804,270 4/1974 Michaud et al. ............. 24/1 CMX
`3,850,313 11/1974 Rackman et al............. 214/1 CMX
`3,888,362
`6/1975 Fletcher ....................... 214/1 CMX
`3,890,552
`6/1975 Devol et al. ......................... 318/.568
`Primary Examiner-Joseph F. Ruggiero
`(57)
`ABSTRACT
`Visual system for determining position in space and/or
`orientation in three-dimensional space for purposes, for
`example, of directing or instructing an industrial robot
`to perform manipulative acts and apparatus employing
`the visual system. The system includes a portable object
`arbitrarily movable in three-dimensional space and pos
`sessing the discernible properties of position in space
`and/or orientation in space. One or more sensors ex
`tract visual information or image data from the portable
`object and convert the same to an electric signal or
`signals. A computer is connected to receive the signal
`or signals which are analyzed and, in the case of the
`industrial robot, the information obtained is used to
`prepare operating instructions.
`61 Claims, 13 Drawing Figures
`
`2A
`
`
`
`ABB Inc. Exhibit 1008, Page 1 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent Mar. 27, 1979
`
`Sheet 1 of 4
`
`4,146,924
`
`
`
`
`
`Fig
`
`KEYBOARD
`
`
`
`PORTABLE
`MEANS
`
`TO
`COMPUTER
`
`2A
`
`Fig.2
`
`hl/
`
`A
`
`12-1
`
`IOB
`
`OA -
`
`hl/
`
`OA
`
`UENCER
`SEQUENCE
`
`FROM
`COMPUTER
`
`ABB Inc. Exhibit 1008, Page 2 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent Mar. 27, 1979
`
`Sheet 2 of 4
`
`4,146,924
`
`
`
`
`
`2
`
`P
`
`A
`2B
`\
`POSSIBILITY
`& - '''
`O
`B
`
`POSSIBILITY NO. 2
`
`2A
`
`TO COMPUTER
`
`Fig.5A
`
`hu
`OA
`
`OC
`
`hv
`21B
`OB
`
`FROM COMPUTER
`
`SEQUENCER
`
`Fig.5C
`
`OA2
`
`Fig.58
`to 2
`O /
`\
`\
`^ - OA
`
`ABB Inc. Exhibit 1008, Page 3 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Mar. 27
`
`s
`
`1979
`
`Sheet 3 of 4
`
`4,146,924
`
`TV
`
`TO SUPPORT
`STRUCTURE
`OF ROBOT
`179
`74
`
`
`
`ABB Inc. Exhibit 1008, Page 4 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent Mar. 27, 1979
`POWER OPERATION
`FIXTURE
`3-N2
`WORKPIECE
`MAGAZINE
`
`on
`
`has
`
`
`
`
`
`
`
`4,146,924
`
`Sheet 4 of 4
`as
`FINISHED
`WORKPIECE
`3
`conveyor
`3OB te.
`pPERATORY-3oc Y 6
`O Yn
`
`M
`
`N
`N
`
`(RSRQTY
`NY SNE
`NUSGS,
`3.
`
`FIXTURE
`
`- - -
`\
`M
`\- - -/
`7oo
`
`---
`
`4.
`/ -5
`
`16,
`CONVEYORA
`- - - - -
`
`12
`MAGAZINE
`-2s.
`Gzoo)
`--- Y - 11
`- - -
`
`
`
`OO
`
`-
`
`ABB Inc. Exhibit 1008, Page 5 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`1.
`
`SYSTEM FOR VISUALLY DETERMINING
`POSITION IN SPACE AND/OR ORIENTATION IN
`SPACE AND APPARATUS EMPLOYNG SAME
`
`4,46,924
`2
`Another object is to permit such instruction or pro
`gramming in a way similar to that used by one human
`instructing another.
`Still another object is to provide a system that points
`out obstacles to the robot,
`A further object is to provide a system to aid in pro
`gramming of other numerically-controlled machine
`such as, for example, numerically-controlled machine
`tools.
`A still further object is to provide such programming
`in a manner that does not endanger a human operator.
`A still further object is to provide a visual system for
`industrial robots and the like that need not be rigidly
`positioned relative to a work surface or space.
`These and still further objects are discussed hereinaf.
`ter and are delineated in the appended claims.
`The foregoing objects are achieved in a visual system
`to determine position in three-dimensional space and/or
`orientation in three-dimensional space, which includes a
`portable object arbitrarily movable in three-dimensional
`space and possessing the discernible properties of posi
`tion in space and/or orientation in space. One or more
`sensors extract visual information or image data from
`the portable object and convert that information to an
`electric signal or signals. A computer, properly pro
`grammed is connected to receive the signal or signals
`from the sensor or sensors and evaluate the same to
`provide an indication of position in three-dimensional
`space and/or orientation in three-dimensional space.
`The computer, on the basis of the information received
`and in the light of its pre-programs, prepares messages
`that may be transmitted to the mechanically-active ele
`ments of one or more robots which, in turn, act or per
`form some function on the basis of the messages re
`ceived.
`The invention is hereinafter described with reference
`to the accompanying drawing in which:
`FIG. 1 is a diagrammatic representation of a system
`employing the present inventive concepts and shows,
`among other things, portable means and a plurality of
`sensors;
`FIG. 2 is a schematic representation, partly block
`diagram in form, showing a portion of the system of
`FIG. 1, the portable means being two lights and the
`sensor being a television camera;
`FIG. 3 is an isometric view of some of the elements
`shown in block diagram form as separate elements in
`FIG. 1 but combined in FIG. 3;
`FIG. 4 is a schematic representation, partly block
`diagram in form, of a modification of the system of FIG.
`2 in that the later figure shows two cameras;
`FIG. 5A shows a single camera as a sensor and porta
`ble means composed of three lights or light sources;
`FIG. 5B shows a cluster of lights which, together,
`form a single light source of a type that can be em
`ployed as any one of the light sources in FIG. 5A;
`FIG. 5C shows an arbitrarily-shaped distributed light
`source that can be used in FIG. 5A;
`FIG. 6A is an isometric, diagrammatic representation
`of a machine that includes the present teachings;
`FIG. 6B is a diagrammatic representation, partly
`block diagram in form, of an actual industrial robot
`system that has been experimented with;
`FIG. 7 shows a plan-view layout for a typical indus
`trial operation for the robot of FIG. 6;
`FIG. 8 is a plan view showing typical visual program
`ming parts with respect to the locations of FIG. 7;
`FIG. 9 shows a fiducial array; and
`
`The present invention relates to systems for determin
`ing position in space and/or orientation in space, that
`may be used, for example, to instruct an industrial robot,
`and the apparatus employing such systems.
`There accompanies herewith a paper by the present
`inventors entitled "Visual Robot Instruction', Proceed
`ings of the Fifth International Symposium on Industrial
`Robots, Chicago, Ill., Sept. 22-24, 1975; said paper is
`hereby incorporated herein by reference. Some, but not
`all, of the material in the paper is included hereinafter.
`Most of the description that follows is centered around
`instructing industrial-type robots, sometimes called pro
`grammable manipulators, but the concepts herein dis
`closed have wider use.
`Industry demands a fast and safe method to program
`20
`industrial robots. Prior proposals for such programming
`include the use of potentiometers, voice control, cath
`ode ray tube (CRT) light pens, master-slave harnesses
`and manual controls, as is noted in said paper. Each
`25
`programming aid offers advantages, but even when
`they are used together they cannot specify what a fore
`man easily tells a human operator. Each aid has distinct
`limitations. Programmers using potentiometers tend to
`move each degree-of-freedom one at a time. Joysticks
`30
`provide some coordination in controlling two or three
`degress-of-freedom, but robots typically have more
`than three. Voice control is a convenient method for
`specifying incremental motion in world or hand coordi
`nates; however, if the increment is numerically speci
`35
`fied, performance is limited in speed by the program
`mer's ability to quantify his visual information. Using
`the voice control technique of moving the robot in slow
`motion until a "Stop' utterance is given may be suscep
`tible to the problem of timing consistency on the part of
`40
`the programmer. Additionally, voice control is not well
`suited to describing complex trajectories. Currently
`voice control is also an expensive programming aid.
`CRT light pens are convenient for specifying points on
`a plane; however specifying points in three dimensional
`45
`space on two screens is difficult. Describing a trajectory.
`is very difficult. The cost of CRTs must also be weighed
`in a decision to include this programming aid. The use
`of a master-slave harness is perhaps the most expensive
`robot programming aid. It is also limited in some ways
`50
`to robots with an anthropoid configuration. For exam
`ple, adjustments of harness elbow elevation to avoid an
`obstacle may not cause a non-anthropomorphic arm to
`adjust appropriately. The programming technique of
`physically leading a robot through a sequence of mo
`55
`tions is limited by safety considerations. Uncontrolled
`behavior is possible if power assist is used.
`When using most programming aids, programmers
`come close to the robot to inspect the location of the
`gripper. This proximity endangers the programmer.
`For safety, it is desirable if a sequence of motions could
`be specified with the programmer in the workspace and
`the robot inactive.
`Accordingly, it is an object of the present invention
`to provide a system adapted to facilitate the instruction
`or programming of numerically-controlled, industrial
`robots to enable such robots to perform new or addi
`tional tasks.
`
`10
`
`15
`
`65
`
`ABB Inc. Exhibit 1008, Page 6 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`15
`
`10
`
`4,146,924
`4.
`3
`structure. This structure enables the two lights to be
`FIG. 10 shows a modification of the portable means
`placed on opposite sides of an object, for example. The
`of FIG. 1.
`center of the two lights would, in this circumstance,
`As will be evident as the story unfolds, what the
`correspond to a position that the programmer wants the
`present inventors have done here is present a surrogate
`center of the robot's gripper (see fingers 70 and 71 in
`for a thing to-be-sensed in three-dimensional space or
`FIG. 6A) to assume. A motor activated by the switch. 8
`for a point, a line, or the like in such space. From the
`surrogate, can be obtained visual information or image
`on the VPD controls the distance between the two
`lights 10A and 10B, as previously discussed. Distance
`data that can be interpreted and then used. The appara
`tus designated 102 in FIG. 1 includes a visual system
`information may be sensed and communicated to the
`101 to determine position in space and/or orientation in
`control computer 3. This information may be used to
`space; in FIG. the surrogate is the portable means
`verify visual computations. The distance between light
`shown at 1. The portable means 1 is arbitrarily movable
`support members 12A and 12B can be made large to
`in three-dimensional space and, as later discussed in
`surround large objects without blocking the line of sight
`detail, possesses the discernible properties of position in
`from the TV camera 2A to the lights 10A and 10B. This
`space and orientation in space. Sensors 2, 2', etc., are
`distance can be made small to avoid interference with
`positioned to extract visual information or data from the
`work station structures. Although programming can be
`portable means 1 and are operable to convert the visual
`done with the workpiece in position, workpiece pres
`information or data to an electric signal or signals. A
`ence is not required by the visual programming tech
`computer 3 is connected to receive the electric signal or
`nique herein disclosed.
`signals from the sensors 2, 2"... and is programmed to
`20
`If the vision system has one television camera, as it
`i. evaluate the same to provide an indication of position in
`has in FIG. 2, the image of a small light can only specify
`space and/or orientation in space on the basis thereof.
`a point in a plane. This plane may be horizontal and just
`The apparatus 102 further includes robots 4A, 4B, etc.,
`above the dominant materials handling surface of the
`whose actions and movements are controlled by the
`workstation. To specify points in space only a single
`computer on the basis of the visual data, as discussed in
`25
`light is necessary. A switch on the keyboard 5A can be
`detail in later paragraphs. For convenience of explana
`pressed to record each point, a numerical keyboard
`tion the visual system 101 in FIG. and later described
`entry may be used to specify how far beyond or inside
`systems 101A and 101B include those elements that
`the reference plane is the three dimensional command
`have to do with locating coordinates in space, the term
`point. This three dimensional approach suffers either
`"robot' designates the mechanical elements that per
`30
`from inaccuracies in estimated distances or a time con
`form some mechanical act, and the term "machine' is
`suming measurement process. For applications where
`used to designate the two combined with some work
`pick and place points are in the same plane, a vision
`space (e.g., a machine tool).
`system with one television camera may suffice. Macros
`The portable means can be one or more lights; in
`such as "go forward to reference plane, close gripper,
`FIG. 2 it is shown at 1A as two individual sources of 35
`back off from reference plane' might be issued with a
`light 10A and 10B. Each source acts effectively as a
`single placement of the VPD 20 and activation of a
`point source of light in the system 101A of FIG. 2. The
`particular switch on the VPD keyboard 5A.
`sensor in FIG. 2 is a television camera 2A (two or more
`If the vision system has two television cameras, as
`cameras can be employed). The lights 10A and 10B can
`shown at 2A and 2B in FIG. 4, the three-dimensional
`be modulated ON-OFF, or their intensity and/or fre
`location of the lights on the VPD may be computed
`quency can be modified by a sequencer 11 (the se
`directly. The accuracy of this computation depends on
`quencer need not be a separate unit; sequencing func
`the vision system. For fixed camera mounts, factors
`tions can be performed by the computer); and the orien
`affecting accuracy include the field of view, spatial
`tation of the lights in space and relative to the camera
`sampling rate, inter-camera geometry, location of lights
`can be changed as now explained with reference to
`45
`relative to television cameras, and camera geometric
`FIG. 3 wherein the elements 1A and a keyboard 5A are
`distortion. To quantify the magnitude of the accuracy
`combined in a single portable unit or visual program
`issue, one can assume a 100 cm line in a reference plane
`ming device (VPD) that is marked 20.
`perpendicular to the optical axis. If this line covers a
`In the device 20, the lights are again designated 10A
`horizontal scan line with 250 samples/line and the light
`and 10B and the distance separating them can be
`50
`activates a single picture element, location is theoreti
`changed by an operator using a control switch 8 which
`cally limited to + 2 mm. For cameras with pan/tilt
`operates an activator (e.g., small electric motor or sole
`mounts, precision servo-mechanisms are important.
`noid not shown). The device 20 is moved about by an
`To describe the proper way to grasp a workpiece, the
`operator who grasps the same at a hand grip 9. Electric
`vector from one robot fingertip to the other fingertip
`power to the device 20 is connected through a cord 6
`55
`must be specified. For this purpose, the design of FIG.
`which serves, as well, to transmit messages between the
`3 is also appropriate. Each light specifies where each
`device 20 and the computer 3.
`robot fingertip (the robot's fingers are marked 70 and 71
`The visual programming device 20 can be used to
`in FIG. 6A and the fingertips are marked 70A and 71A,
`conveniently communicate spatial information to an
`respectively) is to be before a grasping instruction is
`industrial robot to cause the robot to perform some task
`issued. The centroid between fingertips (points in space)
`such as, for example, to grasp an object in its gripper. It
`and orientation between fingertips may be specified at
`is held in a programmer's hand. It has an array of lights,
`the same time. However, if a light is obscured, fingertip
`the lights 10A and 10B, at one end, and a set of switches,
`orientation may be specified elsewhere in the field of
`the keyboard 5A, at the other end. The switches com
`view. Activation of switches on the keyboard 5A may
`municate information to the computer which, as previ
`65
`be used to tell the system how to interpret the visual
`ously noted, activates the robot (or robots). The lights
`instruction. Three types of instruction have been de
`may be under computer control. The VPD 20 shown
`scribed thus far: (1) point in space, (2) orientation be
`has the lights 10A and 10B at the tips of a parallel jaw
`
`
`
`ABB Inc. Exhibit 1008, Page 7 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`10
`
`4,146,924
`5
`6
`tween fingertips, and (3) point in space plus orientation
`the computer to interpret this single light location as
`between fingertips.
`data to specify finger direction relative to the most
`If a VPD with just one light is used, positions and
`recent fingertips direction vector or to the most re
`orientation between fingertips cannot be specified at the
`cently specified point in space.
`same time. However, this can be done in a two-step
`For situations involving a vision system having a
`procedure. Activation of one control switch specifies a
`single television camera, as depicted in FIG. 2, it will
`point in space. Activation of another switch signals the
`often be adequate to make the general specification that
`computer to use the present light position and the most
`robot finger direction is to be perpendicular to the refer
`recently specified position to compute an interfingertips
`ence plane. For other situations the general specifica
`direction vector.
`tion of a vertical finger direction (e.g., horizontal refer
`If the two light VPD in FIG. 3 is used to specify a
`ence plane) will be appropriate.
`unique orientation between a pair of robot fingertips
`Specification of trajectories is useful to avoid obsta
`which are structurally different, a distinction between
`cles, contouring and for special applications such as, for
`the two lights must be made. This distinction is possible
`example, paint spraying, welding or applying adhesives.
`using colored lights, but then the television camera must
`A trajectory is specified by an initial gripper configura
`15
`have provisions for color vision. Another method of
`tion (location and orientation), an ordered set of inter
`distinguishing lights is to turn them on and off sequen
`mediate gripper configurations, a final gripper configu
`tially by the sequence 11 under the proper control of the
`ration and the time for transit between each adjacent
`computer 3. With this scheme, each light is sensed dur
`pair of gripper configurations. The gripper passes
`ing a different frame of the television camera. The time
`through the intermediate gripper configurations with
`20
`required for this technique is limited by turn-on and
`out halting. Servomechanisms to achieve this form of
`turn-off times of the lights and of the photosensitive
`motion are available.
`element of the television camera. Lights may also be
`As described above, a VPD can conveniently specify
`distinguished by size or intensity. Intensity discrimina
`gripper configurations. Thus a VPD is a useful means to
`tion requires a grey scale interface between the televi
`specify trajectories. A number of options are possible to
`25
`sion camera and the computer. Light size does not map
`specify the time for each segment of the trajectory. For
`directly to image size due to distance attenuation and
`many situations, it is appropriate to minimize the time to
`directionality of light sources.
`traverse a trajectory. A command on the keyboard 5A
`A distinction between lights is also useful for the case
`can specify the minimizing function. In this case, inter
`depicted in FIG. 4 wherein it is assumed that the four
`mediate gripper configurations may be passed through
`points of ray intersection marked 21A, 21B, 21C and
`with less accuracy. A second approach is to have key
`21D are in the same plane. In this case, it is not possible
`board entries specify the time for a trajectory segment
`to distinguish from the light image geometry which of
`immediately after specifying the concluding gripper
`the two combinations of positions is actually the case
`configuration for that segment. Rather than use numeri
`without further measures. Even interlight distance in
`35
`cal entries, selection of standard "slow,' "medium' and
`formation will not always lead to a distinction between
`"high' speeds may be preferable. A third approach is to
`the two cases since both possibilities may have the same
`activate a realtime sampling algorithm. In this case the
`interlight distance. The region in space where interlight
`servo system would be instructed to replicate the timing
`distance is the same is a function of the angle between
`of the robot programmer. Keyboard entries could spec
`the optical axes of the two television cameras 2A and
`ify if a fraction or a multiple of measured realtime per
`2B. This region in space is widened by limitations on
`formance was desired for playback.
`estimating distances using stereo vision. To overcome
`The VPD 20 specifies gripper position and orienta
`the ambiguity presented by the configuration of FIG. 4,
`tion. Robot arms with different kinematic and structural
`it is necessary to identify the individual light 10A and
`configurations will have links occupying different por
`10B by, for example, the ON-OFF procedure above
`tions of a workspace for the same gripper position and
`45
`described.
`orientation. Movement toward a particular gripper
`In addition to the vector between robot fingertips
`configuration may cause the problem of contact be
`70A and 71A, a vector pointing in the direction of the
`tween an arm structure and a workstation structure.
`fingers 70 and 71 in FIG. 6A will frequently have to be
`One solution to the problem is to use the VPD to spec
`specified. For example, to make more rigid contact with
`ify a set of points from which an envelope of worksta
`50
`a particular class of workpiece, robot fingers may have
`tion structures can be computed. If such a set of points
`contact surfaces which are not symmetric. Another
`is specified before initiating standard VPD program
`reason for wanting to specify finger direction is to pre
`ming, a gripper configuration command can be refuted
`vent the wrist structure shown at 72 in FIG. 6A from
`in the event that a collision is predicted. The robot
`colliding with workstation structures.
`programmer would be notified of the result so that he
`55
`Finger direction can be specified by a third light in
`can try another gripper configuration.
`the array 1B as shown in FIG. 5 wherein the third light
`For a robot arm with redundant degrees-of-freedom,
`is labeled 10C. Assuming that the three lights can be
`a search can be conducted to select an arm configura
`located and distinguished, a robot finger direction vec
`tion which avoids obstacles. Subsequent refinement of
`tor can be computed which is in the plane defined by
`the search for an arm configuration would have as a
`lights 10A and 10B and the third light 10C. The finger
`goal the stipulation that the configuration be rapidly
`direction vector may be specified as being perpendicu
`attained as an intermediary between the previous grip
`lar to the line between lights 10A and 10B and passing
`per configuration command and the next one. A scheme
`through the third light 10C. Finger direction can also be
`for modeling the outlines of obstacles and arm links is
`specified by using a VPD with a single light, as above
`discussed in one of the references mentioned in said
`65
`noted; in this latter case the light would be located
`paper.
`where the third light is located using the three-light
`The explanation in this and the next several para
`technique. Activation of a switch on the keyboard tells
`graphs is made with respect to FIGS. 6A and 6B. FIG.
`
`30
`
`ABB Inc. Exhibit 1008, Page 8 of 13
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`10
`
`15
`
`4,146,924
`8
`7
`6A illustrates diagrammatically what for present pur
`nique permits the VPD to be used in the absence of a
`robot and provides information which is independent of
`poses is called a machine that is assigned the numeral
`the particular robot employed (see a paper of the pres
`103; the machine 103 consists of a worktable 91 having
`ent inventors, Proceedings of the Third Milwaukee
`a work surface 90, a visual system 101B and a robot 4M;
`Symposium, Milwaukee, Wisc., April 18, 19, 1975, enti
`other mechanical parts of the robot 4M are shown in
`tled "Robots with TV: Attaching Robots to Machines
`FIG. 6B. In a sense, the whole of the elements in FIGS.
`Thru Software" that accompanies herewith, which
`6A and 6B could be called a robot, but as used herein
`paper is hereby incorporated herein by reference). It
`and as above noted, the term "robot' in all variations
`should be noted here that the fiducial array 50 can have
`embraces the mechanical and electro-mechanical parts
`more than the three lights in FIG. 9 and each can take
`shown in some detail in FIG. 6B. Also, strictly speak
`the form of the cluster 10A, in FIG. 5B; or the array 50
`ing, the work surface 90 need not be part of a machine
`can be, in some instances, a plurality of distributed
`of which the robot 4M is also a part, but it can be; and
`sources of the type shown in FIG. 5C. Also, the fiducial
`it is assumed such for this explanation. Anyway, the
`array need not be at any particular position relative to
`cylindrical object labeled 60 in FIG. 6A is located or
`or on the surface 90; what is essential is that the fiducial
`found within the working volume of the machine and,
`means 50 bear some known or determinable spatial
`more precisely, its x-y coordinates are determined (its
`relationship to the work surface 90 or other work space
`external dimensions are determinable, as well) by inter
`so that once the fiducial means is found by the camera
`action between the VPD 20, the camera 2A and the
`2Athen the location of all other parts of the work space
`computer 3, as now explained.
`can be accomplished.
`Various x-y coordinates on the surface 90 are located
`20
`The diagrammatic representation of the robot 4M in
`with respect to fiducial or reference means 50 that com
`FIG. 6A is almost self-explanatory, but some features
`prises a plurality of lights 51A and 51B (there may also
`are pointed out here. Movement of the arm 73 in the x,
`be three lights as shown in FIG. 9, or more, and each
`y and z directions is effected by lead screws 75A, 75B
`elements 51A ... need not be a light, nor a single light).
`and 75C, respectively, that are rotated by d-c servos
`The important matter here is that the location of the
`25
`76A, 76B and 76C (not shown) respectively. The robot
`fiducial means 50 be fixed relative to the workable sur
`is mounted on a base 79 and ways 78A1..., 78B1... and
`face 90 or be determinable relative thereto. In FIG. 6A
`78C1.... Position information is obtained from coded
`the TV camera 2A is attached to a camera arm 74. The
`discs 77A... using incremental angle encoder pickup
`camera arm 74 and the arm 73 of the robot 4M are
`counters (not shown). The wrist portion 72 of the arm
`assumed for present purposes to bear a known spatial
`30
`73 contains a stepping motor within the block marked
`relationship relative to one another and the computer 3
`72A to effect rotation of the fingers and a solenoid and
`is pre-programmed to be aware of the relationship. The
`cable actuator moves the fingers 70 and 71 toward each
`TV camera 2A is situated so as to be able to view the
`other. As above noted, the x-y position of the TV cam
`lights 51A, 51B, etc. in the same scene or separate
`era 2A and the fingers 70 and 71 are indexed to one
`scenes through pan and tilt motion of the camera. The
`35
`another so that a coordinate determined by the camera
`visual location of the lights 51A, 51B, etc. is noted.
`can be found by the fingers. In fact, the camera arm 74
`These data are interpreted by the computer 3 to yield
`is attached to the robot base structure 79 and its position
`x-y coordinates of the fiducial means 50. In the same
`in space thus is determinable. A detailed description
`way, the location of the object 60 on the work surface
`now follows of a task that can be performed by an in
`90 is achieved using the VPD 20 in the manner previ
`dustrial robot, like the robot 4M.
`ously indicated. The TV camera 2A visually locates the
`A "pick and place' task may be described by a human
`VPD lights 10A, 10B. The computer 3 interprets these
`instructor to a human operator as follows (refer to FIG.
`data to yield the x-y position of the object which is
`7):
`determined relative to the fiducial means 50. The robot
`Take a part from this magazine and place it in that
`4M has an end effector or wrist structure 72 comprising
`45
`fixture like this. Next press both buttons at the same
`the fingers 70 and 71, as previously mentioned. The
`time to enable the power operation. Then remove the
`computer 3 also extracts orientation information from
`finished part from the fixture and place it on the con
`the VPD data which permit the wrist structure 72 to be
`veyor like so. Now you are ready to repeat the cycle.
`aligned for purposes of grasping the object 60 with the
`Let it be supposed now that a workstation, such as is
`fingers 70 and 71. The end purpose here is to have the
`shown in FIG. 7, is to be used by a robot such as the
`robot 4M grasp the object 60 in the fingers 70 and 71
`robot 4M. Because modifying the safety features of a
`and, say, move it to another location in the work space
`workstation is not good practice, the dual pushbuttons
`or volume.
`may be operated by a single arm robot if a pushbar is
`The robot 4M can independently move the end effec
`tor 72 along each of three mutually orthogonal axes,
`attached as shown in FIG. 7. Such a bar could serve to
`satisfy the anti-repetition logic of the power device and
`one of which is perpendicular to the worktable surface
`also guarantee that the robot is safe. Furthermore, resto
`90. The robot 4M can also rotate the end effector 72
`ration for human operation is simple.
`about an axis which is perpendicular to the worktable
`Task locations are shown crosshatched in FIG. 7;
`surface 90. The fingers 70 and 71 on the end effector 72
`they are marked 30A, 30B and 30C for a workpiece 13
`can be closed and opened to permit the robot to grasp
`and 31 for a pushbar 14. The robot proceeds from the
`and release the workpiece 60. The fiducial array 50 is
`"home' position marked 000 in FIG. 8, takes the next
`rigidly attached to the worktable surface 90; the array
`workpiece labeled 13 (at location 30A) from the maga
`50 defines a vision/robot system invariant measurement
`zine shown at 12 and places it in the center of the fixture
`coordinate system. The position and orientation infor
`marked 15. Then the robot goes to the center of the
`mation conveyed by the VPD 20 through the light
`65
`pushbar, station 31 of the pushbar 14, and depresses it.
`sources 10A and 10B, as sensed by the TV camera 2A
`Next the robot removes the workpiece 13 from the
`and extracted by the computer 3, is measured relative to
`fixture 15 (location 30B) and