I 1111111111111111 11111 1111111111 lllll 111111111111111 111111111111111 IIII IIII
`US008095237B2
`
`c12) United States Patent
`Habibi et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,095,237 B2
`Jan.10,2012
`
`(54) METHOD AND APPARATUS FOR SINGLE
`IMAGE 3D VISION GUIDED ROBOTICS
`
`(75)
`
`Inventors: Babak Habibi, North Vancouver (CA);
`Simona Pescaru, North Vancouver
`(CA); Mohammad Sameti, Coquitlam
`(CA); Remus Florine! Boca, North
`Vancouver (CA)
`
`(73) Assignee: RoboticVISIONTech LLC, Great Falls,
`VA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 525 days.
`
`(21) Appl. No.: 10/634,874
`
`(22) Filed:
`
`Aug. 6, 2003
`
`(58) Field of Classification Search .................. 348/287,
`348/291, 154, 190, 42, 94,552, 191; 901/14,
`901/17, 46--47, 6; 414/730, 737; 382/154,
`382/103, 106, 276, 293, 295, 298; 700/245-246,
`700/257-259, 253, 279; 29/407.04, 714,
`29/407.1, 702,720,218; 318/568.15, 568.13,
`318/568.16, 640
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,986,007 A
`10/1976 Ruoff, Jr.
`4,011,437 A
`3/1977 Hohn ....................... 235/151.11
`4,146,924 A *
`3/1979 Birk et al ...................... 700/259
`4,187,454 A
`2/1980 Ito et al ......................... 318/568
`4,219,847 A
`8/1980 Pinkney et al.
`4,294,544 A
`10/1981 Altschuler et al. ............ 356/376
`4,305,130 A *
`12/1981 Kelley et al. .................. 700/259
`(Continued)
`
`(65)
`
`Prior Publication Data
`
`US 2004/0172164Al
`
`Sep.2,2004
`
`DE
`
`FOREIGN PATENT DOCUMENTS
`19515949 Al
`11/1996
`(Continued)
`
`OTHER PUBLICATIONS
`
`Wei et al., Multisensory visual servoing by a neural network, 1999,
`IEEE, p. 276-280.*
`
`(Continued)
`James Trammell
`Primary Examiner -
`Assistant Examiner - McDieunel Marc
`(74) Attorney, Agent, or Firm - Seed IP Law Group PLLC
`ABSTRACT
`(57)
`A method of three-dimensional object location and guidance
`to allow robotic manipulation of an object with variable posi(cid:173)
`tion and orientation using a sensor array which is a collection
`of one or more sensors capable of forming a single image.
`28 Claims, 7 Drawing Sheets
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 10/153,680,
`filed on May 24, 2002, now Pat. No. 6,816,755.
`
`(30)
`
`Foreign Application Priority Data
`
`Jan. 31, 2002
`
`(CA) ...................................... 2369845
`
`(51)
`
`Int. Cl.
`(2006.01)
`G06F 19100
`(52) U.S. Cl. ........ 700/245; 700/246; 700/253; 700/257;
`700/258; 700/259; 318/568.13; 318/568.15;
`318/568.16; 318/640; 29/702; 29/714; 29/720;
`901/6; 901/14; 901/17; 901/46; 901/47
`
`D
`
`22
`
`ABB Inc. Exhibit 1001, Page 1 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`US 8,095,237 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`4,334,241 A
`6/1982
`Kashioka et al.
`4,402,053 A
`8/1983
`Kelley et al
`4,437,114 A *
`3/1984
`...................... 348/94
`LaRussa
`4,523,809 A
`Taboada et al ................ 350/163
`6/1985
`Corby, Jr. et al
`4,578,561 A
`........ 219/124.34
`3/1986
`4,613,942 A *
`Chen .
`. .............. 700/259
`9/1986
`4,654,949 A *
`Pryor ......................... 29/407.04
`4/1987
`4,687,325 A
`Corby, Jr ........................... 356/1
`8/1987
`4,791,482 A
`Barry et al
`................... 358/107
`12/1988
`4,835,450 A
`Suzuki
`................ 318/568.13
`5/1989
`4,871,252 A *
`Beni et al ...................... 356/457
`10/1989
`4,879,664 A
`Suyama et al.
`............... 364/518
`11/1989
`4,942,539 A *
`McGee et al.
`. ............... 700/253
`7/1990
`4,985,846 A
`Fallon ........................... 364/513
`1/1991
`5,083,073 A
`Kato ............................. 318/577
`1/1992
`5,160,977 A
`Utsumi ......................... 356/376
`11/1992
`5,208,763 A
`Hong et al ............... 364/551.02
`5/1993
`5,212,738 A
`Chande et al.
`5/1993
`5,300,869 A *
`Skaar et al. ................... 318/587
`4/1994
`5,325,468 A
`Terasaki et al. ................. 395/97
`6/1994
`5,350,269 A
`Azuma et al. ................. 414/416
`9/1994
`5,446,835 A
`Iida et al. ...................... 395/133
`8/1995
`5,454,775 A
`Cullen et al. .................... 483/16
`10/1995
`5,461,478 A
`Sakakibara et al ........... 356/375
`10/1995
`5,499,306 A
`Sasaki et al. . . . . . . . . . . . . . . . . . . 3 82/291
`3/1996
`5,521,830 A
`Saito ........................ 364/474.34
`5/1996
`5,568,593 A
`Demarest et al. ............... 395/82
`10/1996
`5,608,818 A
`Chini et al .................... 382/153
`3/1997
`5,633,676 A *
`Harley et al
`................... 348/95
`5/1997
`5,696,673 A *
`Pryor
`..................... 700/66
`12/1997
`5,715,166 A
`Bes! et al. ................ 364/474.24
`2/1998
`5,745,523 A
`Dent et al.
`.................... 375/216
`4/1998
`5,784,282 A
`Abitbol et al. ........... 364/474.28
`7/1998
`5,802,201 A
`Nayar et al.
`.................. 382/153
`9/1998
`5,809,006 A
`Davis et al ................. 369/275.4
`9/1998
`5,870,527 A
`Fujikawa et al.
`. .............. 395/80
`2/1999
`5,956,417 A *
`Pryor
`............... 382/154
`9/1999
`5,959,425 A *
`Bieman et al. ........... 318/568.15
`9/1999
`5,974,169 A
`Bachelder . . . . . . . . . . . . . . . . . . . . . 3 82/ l 5 l
`10/1999
`5,978,521 A
`Wallack et al.
`11/1999
`6,004,016 A
`Spector .................... 364/167.02
`12/1999
`6,044,183 A *
`Pryor
`......................... 382/287
`3/2000
`6,064,759 A
`Buckley et al
`............... 382/154
`5/2000
`6,081,370 A *
`Spink .
`. ................ 359/369
`6/2000
`6,115,480 A
`Washizawa
`9/2000
`6,141,863 A *
`...................... 29/714
`Hara et al.
`11/2000
`6,167,607 Bl*
`Pryor ......................... 29/407.04
`1/2001
`6,211,506 Bl
`Pryor et al.
`4/2001
`6,236,896 Bl
`Watanabe et al.
`5/2001
`............ 700/250
`6,278,906 Bl
`Piepmeier et al
`8/2001
`6,301,763 Bl*
`........... 29/407.04
`Pryor
`10/2001
`6,341,246 Bl
`Gerstenberger et al ....... 700/245
`1/2002
`Holec .......................... 356/4.03
`6,392,744 Bl
`5/2002
`Watanabe et al. ............. 700/253
`6,463,358 Bl
`10/2002
`6,466,843 Bl
`Bonanni et al. ............... 700/245
`10/2002
`Beiman ......................... 382/153
`6,490,369 Bl
`12/2002
`Bachelder et al ............. 382/181
`6,516,092 Bl
`2/2003
`6,529,627 Bl
`Callari et al.
`................. 382/154
`3/2003
`6,546,127 Bl
`Seong et al.
`.................. 382/152
`4/2003
`6,549,288 Bl
`Migdal et al. ................. 356/603
`4/2003
`6,580,971 B2
`Bunn et al.
`.................. 700/259
`6/2003
`6,594,600 Bl *
`Arnoul et al. ................... 702/94
`7/2003
`6,628,819 Bl
`Huang et al
`9/2003
`6,721,444 Bl
`.................... 382/154
`Gu et al.
`4/2004
`6,724,930 Bl
`Kosaka et al.
`................ 382/154
`4/2004
`6,741,363 Bl
`Kaupert ........................ 356/602
`5/2004
`6,748,104 Bl
`Bachelder et al. . . . . . . . . . . . . 3 82/ l 5 l
`6/2004
`6,754,560 B2 *
`Fujita et al. ................... 700/245
`6/2004
`6,804,416 Bl
`Bachelder et al. . . . . . . . . . . . . 3 82/294
`10/2004
`6,816,755 B2 *
`Habibi et al.
`................. 700/259
`11/2004
`6,836,702 Bl
`Brogardh et al
`............. 700/245
`12/2004
`6,853,965 B2 *
`Massie et al.
`................... 703/6
`2/2005
`6,970,802 B2
`Ban et al. ...................... 702/153
`11/2005
`7,006,236 B2
`Tomasi et al. ................. 356/614
`2/2006
`7,009,717 B2
`Van Coppenolle et al. ... 356/603
`3/2006
`7,024,280 B2 *
`Parker et al.
`. ... 700/245
`4/2006
`7,061,628 B2
`Franke et al.
`6/2006
`7,084,900 Bl
`Watanabe et al. ............... 348/94
`8/2006
`
`7,177,459 Bl
`7,693,325 B2 *
`2001/0034481 Al*
`2001/0055069 Al*
`2002/0019198 Al
`2002/0028418 Al*
`2002/0156541 Al
`2002/0159628 Al *
`2003/0004694 Al *
`2003/0007159 Al
`2003/0182013 Al
`2003/0202691 Al *
`2004/0037689 Al
`2004/0041808 Al
`2004/0073336 Al
`2004/0081352 Al
`2004/0114033 Al
`2004/0172164 Al
`2004/0193321 Al
`2004/0233461 Al*
`2005/0002555 Al
`2005/0097021 Al
`2005/0126833 Al
`2005/0233816 Al
`2005/0246053 Al
`2005/0273202 Al
`2006/0025874 Al
`2006/0088203 Al
`2006/0119835 Al
`2006/0210112 Al
`2007 /0073439 Al
`2010/0040255 Al *
`
`2/2007 Watanabe et al. ............. 382/151
`4/2010 Pulla et al. .................... 382/154
`10/2001 Horn ............................. 600/407
`12/2001 Hudson ......................... 348/302
`2/2002 Kamono ......................... 451/41
`3/2002 Farag et al
`..................... 433/29
`10/2002 Yutkowitz ....................... 700/28
`10/2002 Matusik et al ................ 382/154
`1/2003 Aliaga et al. ...................... 703/2
`1/2003 Franke et al.
`................ 700/145
`9/2003 Moreas et al.
`10/2003 Beardsley ..................... 382/154
`2/2004 Watanabe et al. ............. 414/730
`3/2004 Ban et al. ...................... 345/419
`4/2004 Huang et al. .................. 700/245
`4/2004 Ban et al.
`................... 382/154
`6/2004 Eian et al ........................ 348/42
`9/2004 Habibi et al.
`9/2004 Anfindsen et al. ............ 700/257
`11/2004 Armstrong et al ............ 356/620
`1/2005 Kumiya et al
`............... 382/153
`5/2005 Behr et al.
`................... 705/36
`6/2005 Takenaka et al.
`180/8.1
`10/2005 Nishino et al. ................ 473/131
`11/2005 Endou et al. .................. 700/188
`12/2005 Bischoff
`2/2006 Huffington et al .............. 700/44
`4/2006 Boca et al. .................... 382/153
`6/2006 Rastegar et al
`................ 356/28
`9/2006 Cohen et al.
`................ 382/103
`3/2007 Habibi et al.
`................. 700/213
`2/2010 Rhoads ......................... 382/100
`
`DE
`DE
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`
`FOREIGN PATENT DOCUMENTS
`102 36 040
`2/2004
`10319253 Al
`12/2004
`0 114 505
`* 8/1984
`0114505 Al
`8/1984
`0151417
`8/1985
`0493612
`7/1992
`0763406
`3/1997
`0763406 Al
`3/1997
`0911603 Bl
`4/1999
`0951968
`10/1999
`1 043 126
`10/2000
`1 043 642
`10/2000
`1 043 689
`10/2000
`1172183 A2
`1/2002
`1345099 A2
`9/2003
`1 484 716
`12/2004
`63288683
`11/1988
`01124072
`5/1989
`401124072
`5/1989
`07311610
`11/1995
`10049218
`2/1998
`2000024973
`1/2000
`2002018754
`1/2002
`9806015
`2/1998
`0106210 Al
`1/2001
`2005/074653
`8/2005
`
`OTHER PUBLICATIONS
`
`Kim, Computer vision assisted virtual reality calibration, 1999,
`Internet, p. 450-464.*
`Bejczy, Challenges of human-robot communication in telerobotics,
`1996, IEEE, p. 1-8.*
`Sharma, Visual servoing with independently controlled cameras
`using a learned invariant representation, 1998, IEEE, p. 3263-3268. *
`Crowley et al., Maintaining stereo calibration by tracking image
`.
`.
`points, 1993, IEEE, pp. 483-488. *
`Chandrashaker et al., Robotics and structural dynamics charactenza(cid:173)
`tion of the space station remote manipulator system using
`.
`photograrnmetric techniques, 1994, IEEE, pp. 691-694.*
`Triboulet et al., Finding 3D polyhedral object attitude usmg a virtual
`model for industrial machining, 2001, IEEE, pp. 57-65.*
`Zhaohui et al., Image registration using a 3-D scene representation,
`1998, IEEE, pp. 828-832.*
`
`ABB Inc. Exhibit 1001, Page 2 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`US 8,095,237 B2
`Page 3
`
`Safaee-Rad et al., Three-dimensional location estimation of circular
`features for machine vision, 1992, IEEE, pp. 624-640. *
`Safaee-Rad et al., An analytical method for the 3D-location estima(cid:173)
`tion of circular features for an active-vision system, IEEE, pp. 215-
`220.*
`Xu et al., Determining Camera Intrinsic and Extrinsic Parameters
`from Multiple Images of Multiple Balls, 2002, Internet, pp. 1-5.*
`Roger Y. Tsai and Reimar K. Lenz, "A New Technique for Fully
`Autonomous and Efficient 3D Robotics Hand/Eye Calibration",
`IEEE Transactions on Robotics and Automation, vol. 5, No. 3, Jun.
`1989 p. 345.
`Zhengyou Zhang, "A Flexible New Technique for Camera Calibra(cid:173)
`tion",
`http:/ /research .micro soft .corn/scripts/pubs/view. asp ?TR_
`ID~MSR-TR-98-71.
`Guo-Qing Wei, Klaus Arbter, and Gerd Hirzinger, "Active Self-Cali(cid:173)
`bration of Robotic Eyes and Hand-Eye Relationship with Model
`Identification", IEEE Transactions on Robotics and Automation, vol.
`14, No. 1, Feb. 1998 p. 158.
`Yuncai Liu, Thomas S. Huang and Olivier D. Faugeras, "Determina(cid:173)
`tion of Camera Location from 2-D to 3-D Line and Point Correspon(cid:173)
`dences", IEEE Transactions on Pattern Analysis and Machine Intel(cid:173)
`ligence, vol. 12, No. 1, Jan. 1990 p. 28.
`Chien-Ping Lu, Gregory D. Hager and Eric Mjolsness, "Fast and
`Globally Convergent Pose Estimation from Video Images", IEEE
`Transactions on Pattern Analysis and Machine Intelligence, vol. 22,
`No. 6, Jun. 2000 p. 610.
`Thomas Huang, Alfred M. Bruckstein, Robert J. Holt, and Arun N.
`Netravali, "Uniqueness of3D Pose Under Weak Perspective: A Geo(cid:173)
`metrical Proof', IEEE Transactions on Pattern Analysis and Machine
`Intelligence, vol. 17, No. 12, Dec. 1995 p. 1220.
`U.S. Appl. No. 60/413,180, filed Sep. 23, 2002, Eian et al.
`Denzler, J., et al., "Learning, Tracking and Recognition of 3D
`Objects," Proceedings of the IEEEIRSJIGI International Conference
`on Intelligent Robots and Systems, no date available.
`"Advanced Robotic Systems and the Real World," Proceedings of the
`International Conference on Intelligent Robots and Systems (!ROS),
`XP000512662, vol. 1, Munich, Sep. 12, 1994, pp. 89-96.
`W. Meyer, "One-Eyed Robots With 3D Vision," ISRA Vision Sys(cid:173)
`tems AG, Press News, Release No. 16, Jun. 15, 2004, pp. 1-7.
`"3D Robot Guidance with a Single Camera," ISRA Vision Systems
`AG, no date available.
`3D Vision with One Camera, URL~http://neu.isravision.com/
`likecms/index.php?site~site.htrnl&diFisra&nav~ 162,
`download
`date Apr. 12, 2005.
`U.S. Appl. No. 60/587,488, filed Jul. 14, 2004, Boca et al.
`Borotschnig, H., et al., "Appearance-Based Active Object Recogni(cid:173)
`tion," Image and Vision Computing, 18:715-727, 2000.
`Chen, L., et al., "An investigation on the accuracy of three-dimen(cid:173)
`sional space reconstruction using the Direct Linear Transformation
`Technique," J. Biomechanics 27(4):493-500, 1994.
`Denzler J., et al., "Learning, Tracking and Recognition of 3D
`Objects," Proceedings of the IEEE/RSJ/GI, International Conference
`on Intelligent Robots and Systems (IROS) 1: 89-96, 1994.
`Ji, Q., et al., "An Integrated Linear Technique for Pose Estimation
`from different Geometric Features," International Journal of Pattern
`Recognition and Artificial Intelligence, 13(5):705-733, Aug. 1999.
`Jia, Y-B., et al., "Sensing Polygon Poses by Inscription," Proc. of
`1994 IEEE International Conference on Robotics and Automation,
`pp. 1642-1649, May 1994.
`Kim, W.S., "Computer vision assisted virtual reality calibration,"
`URL~http:www.robotics.jpl.nasa.gov/publications/ Won Kirn/
`ra98_vrc.pdf., pp. 1-14, 1999.
`Kovacic, S., et al., "Planning Sequences of Views for 3-D Object
`Recognition and Pose Determination," Pattern Recognition
`31(10):1407-1417, 1998.
`Sanchez, A., et al., "Robot-arm Pick and Place Behavior Program(cid:173)
`ming System Using Visual Perception," in Proceedings of the 15th
`International Conference on Pattern Recognition 4:507-510, Sep.
`2000.
`Tsai, R. et al., "An Efficient and Accurate Camera Calibration Tech(cid:173)
`nique for 3D Machine Vision," Proceedings ofIEEE, Conference on
`Computer Vision and Pattern Recognition, pp. 364-374, 1986.
`
`Tsai, R., "A Versatile Camera Calibration Technique for High-Accu(cid:173)
`racy 3D Machine Vision Metro logy Using Off-theShelfTV Cameras
`and Lenses,"
`IEEE Journal of Robotics and Automation,
`RA-3(4):323-344, Aug. 1987.
`Roger Y. Tsai and Reimar K. Lenz, "A New Technique for Fully
`Autonomous and Efficient 3D Robotics Hand/Eye Calibration",
`IEEE Transactions on Robotics and Automation, vol. 5, No. 3, Jun.
`1989 p. 345.
`Guo-Qing Wei, Klaus Arbter, and Gerd Hirzinger, "Active Self-Cali(cid:173)
`bration of Robotic Eyes and Hand-Eye Relationship with Model
`Identification", IEEE Transactions on Robotics and Automation, vol.
`14, No. 1, Feb. 1998 p. 158.
`Yuncai Liu, Thomas S. Huang and Olivier D. Faugeras, "Determina(cid:173)
`tion of Camera Location from 2-D to 3-D Line and Point Correspon(cid:173)
`dences", IEEE Transactions on Pattern Analysis and Machine Intel(cid:173)
`ligence, vol. 12, No. 1, Jan. 1990 p. 28.
`Chien-Ping Lu, Gregory D. Hager and Eric Mjolsness, "Fast and
`Globally Convergent Pose Estimation from Video Images", IEEE
`Transactions on Pattern Analysis and Machine Intelligence, vol. 22,
`No. 6, Jun. 2000 p. 610.
`Thomas Huang, Alfred M. Bruckstein, Robert J. Holt, and Arun N.
`Netravali, "Uniqueness of3D Pose Under Weak Perspective: A Geo(cid:173)
`metrical Proof', IEEE Transactions on Pattern Analysis and Machine
`Intelligence, vol. 17, No. 12, Dec. 1995 p. 1220.
`Nelson et al., "Robotic Visual Servoing and Robotic Assembly
`Tasks," IEEE Robotics & Automation Magazine 3 (2):23-31, 1996.
`Zhang, "A Flexible New Technique for Camera Calibration," Tech(cid:173)
`nical Report MSR-TR-98-71, Microsoft Corporation, Dec. 2, 1998,
`22 pages.
`U.S. Appl. No. 11/754,218, filed May 25, 2007, Habibi et al.
`U.S. Appl. No. 11/779,812, filed Jul. 18, 2007, Habibi.
`U.S. Appl. No. 11/833,187, filed Aug. 2, 2007, Boca et al.
`U.S. Appl. No. 11/857,680, filed Sep. 19, 2007, Boca et al.
`U.S. Appl. No. 60/719,765, filed Sep. 23, 2005, Habibi et al.
`U.S. Appl. No. 60/808,903, filed May 26, 2006, Habibi et al.
`U.S. Appl. No. 60/832,356, filed Jul. 20, 2006, Habibi.
`U.S. Appl. No. 60/845,932, filed Sep. 19, 2006, Boca et al.
`U.S. Appl. No. 60/875,073, filed Dec. 16, 2006, Beis et al.
`Aloimonos, J., et al., "Active Vision," IEEE, pp. 35-54, 1987.
`Anonymous, "Precise Length Measurement at VSM (Vereinigte
`Schmirgel-Und Maschinen-Fabriken AG)," News of the Astech
`GmbH 2001(4), Dec. 31, 2001.
`Bajcsy, R., "Active Perception," Proc. IEEE 76(8):996-1005, 1988.
`Bakhtari, A., et al., "Active-Vision Based Multi-Sensor Surveil(cid:173)
`lance-An Implementation," IEEE Transactions on Systems, Man
`and Cybernetics, Part C: Applications and Reviews, 36(5):668-680,
`Sep. 2006.
`Borotschnig, H., et al., "Active Object Recognition in Parametric
`Eigenspace," Proc. 9th British Machine Vision Conference, pp. 629-
`638, 1998.
`Bouthemy, P., "A Maximum Likelihood Framework for Determining
`Moving Edges," IEEE Trans. Pattern Analysis and Machine Intelli(cid:173)
`gence 11(5):499-511, 1989.
`Brown, M., et al., "3D Model Acquisition by Tracking 2D
`Wireframes," Proc. 11th British Machine Vision Conference, pp.
`656-665, 2000.
`Callari, F., et al., "Active Recognition: Using Uncertainty to Reduce
`Ambiguity," Proc. 13th International Conference on Pattern Recog(cid:173)
`nition, pp. 925-929, 1996.
`Chaumette, F., "Potential Problems of Stability and Convergence in
`Image-Based and Position-Based Visual Servoing," in D. Kriegman,
`G. Hager and S. Morse, editors, The Confluence of Vision and Con(cid:173)
`trol, vol. 237 of Lecture Notes in Control and Information Sciences,
`pp. 66-78, Springer-Verlag, 1998.
`Chaumette, F., et al., "Structure from Controlled Motion," IEEE
`Trans. Pattern Analysis and Machine Intelligence 18(5):492-504,
`May 1996.
`Constantinescu, D., et al., "Smooth and Time-Optimal Trajectory
`Planning for Industrial Manipulators along Specified Paths," Journal
`of Robotic Systems 17(5):233-249, 2000.
`Corke, P.I., et al., "A New Partitioned Approach to Image-Based
`Visual Servo Control," IEEE Trans. On Robotics and Automation
`17(4):507-515, Aug. 2001.
`
`ABB Inc. Exhibit 1001, Page 3 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`US 8,095,237 B2
`Page 4
`
`Deguchi, K., "Optimal Motion Control for Image-Based Visual
`Servoing by Decoupling Translation and Rotation," in Proc. Int. Conj
`Intelligent Robots and Systems, pp. 705-711, Oct. 1998.
`Drummond, T., et al., "Real-time Tracking of Complex Structures
`with On-line Camera Calibration," Proceedings of British Machine
`Vision Conference, pp. 574-583, 1999.
`Eriksen, R.D., et al., "Classification and pose estimation of uncon(cid:173)
`strained 3D objects," 34th International Symposium on Robotics,
`Jun. 2-5, 2003, 7 pages.
`Hartley et al., Multiple View Geometry in Computer Vision, Cam(cid:173)
`bridge University Press, Cambridge, 2000.
`Holliday et al., "Demonstration of Automated Robotic Workcell for
`Hazardous Waste Characterization," Proceedings of the Interna(cid:173)
`tional Conference on Robotics and Automation 3( 10):788-794, May
`2, 1993.
`Hutchinson, S.A., et al., "Planning Sensing Strategies in a Robot
`Work Cell with Multi-Sensor Capabilities," IEEE Trans. Robotics
`and Automation 5(6):765-783, 1989.
`Hutchison et al., "A Tutorial on Visual Servo Control," IEEE Trans(cid:173)
`actions on Robotics and Automation 12(5), Oct. 1996.
`Lowe, D.G., "Robust Model-based Motion Tracking Through the
`Integration of Search and Estimation," International Journal of Com(cid:173)
`puter Vision 8(2): 113-122, 1992.
`Macfarlane, S., et al., "Jerk-Bounded Manipulator Trajectory Plan(cid:173)
`ning. Design for Real-Time Applications," IEEE Trans. Robotics and
`Automation 19(1):42-52, 2003.
`Marjanovic et al., "Self-Taught Visually-Guided Pointing for a
`Humanoid Robot," XP-002220054, pp. 35-44, 1996.
`Miura, J., et al., "Task-Oriented Generation ofVisual Sensing Strat(cid:173)
`egies in Assembly Tasks," CMU School of Computer Science Tech
`Report CMU-CS-95-116, 42 pages, 1995.
`Motai, Y., et al., "Concatenate Feature Extraction for Robust 3D
`Elliptic Object Localization," Proc. Of the 19th ACM Symposium on
`Applied Computing, pp. 21-28, 2004.
`Murase, H., et al., "Visual Learning and Recognition of3-D Objects
`from Appearance," International Journal of Computer Vision
`14(1):5-24, 1995.
`Naish, M.D., et al., "Coordinated dispatching of proximity sensors
`for the surveillance of manoeuvring targets," Journal of Robotics and
`Computer Integrated Manufacturing 19(3):283-299, 2003.
`
`Nomura et al., "Integrated Visual Servoing System to Grasp Indus(cid:173)
`trial Parts Moving on Conveyer by Controlling 6D0F Arm," Systems,
`Man, and Cybernetics, 2000 IEEE International Conference 3: 1768-
`1775, Oct. 2000.
`Papanikolopoulos et al., "Six Degree-of-Freedom Hand/Eye Visual
`Tracking with Uncertain Parameters," IEEE Transactions on Robot(cid:173)
`ics and Automation 11(5):725-732, Oct. 1995.
`Sang, T., et al., "Vision Assisted Robotic Tele-training," IEEE Inter(cid:173)
`national Conference on Mechatronics and Automation, pp. 1288-
`1293, Jul. 2005.
`Tarabanis, K.A., et al., "A Survey of Sensor Planning in Computer
`Vision," IEEE Transactions on Robotics and Automation 11(1):86-
`104, Feb. 1995.
`Tarabanis, K.A., et al., "The MVP Sensor Planning System for
`Robotic Vision Tasks," IEEE Transactions on Robotics and Automa(cid:173)
`tion 11(1):72-85, Feb. 1995.
`Vincze, M., "Robust tracking of ellipses at frame rate," The Journal
`of Pattern Recognition 34(2):487-498, 2001.
`Whaite, P., et al., "From Uncertainty to Visual Exploration," IEEE
`Transactions on Pattern Analysis and Machine Intelligence
`13(10):1038-1049, 1991.
`Wilson et al., "Relative End-Effector Control Using Cartesian Posi(cid:173)
`tion Based Visual Servoing," IEEE Transactions 12(5):684-696,
`1996.
`Zhou, F., et al., "Complete calibration of a structured light stripe
`vision sensor through planar target of unknown orientations," Image
`and Vision Computing 23:59-67, 2005.
`Office Action for U.S. Appl. No. 111183 ,228, 12 pages, mail date Dec.
`6, 2006.
`Amendment filed in U.S. Appl. No. 11/183,228, 24 pages., Apr. 5,
`2007.
`Office Action for U.S. Appl. No. 11/183,228, 14 pages, Mail Date
`Jun. 8, 2007.
`Amendment filed in U.S. Appl. No. 11/183,228, 8 pages, Mail Date
`Sep. 4, 2007.
`Wilson, et al., "Relative End-Effector Control Using Cartesian Posi(cid:173)
`tion Based Visual Servoing," IEEE Transactions on Robotics and
`Automation, 12(5):684-696, Oct. 1996.
`"3D Robot Guidance with a Single Camera," ISRA Vision Systems
`AG, Machine Vision for Robot Guidance, Nashville, Tennessee, Oct.
`5, 2004.
`
`* cited by examiner
`
`ABB Inc. Exhibit 1001, Page 4 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 1 of 7
`
`US 8,095,237 B2
`
`•
`
`(!) -LL
`
`ABB Inc. Exhibit 1001, Page 5 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 2 of 7
`
`US 8,095,237 B2
`
`l 6
`
`Camera Space 26
`~
`
`Tool Space 28
`
`14
`
`-r V
`
`/,: u,,,,
`
`0
`
`Training Space 32
`
`22
`
`Robot Base Space 30
`
`Fig. 2
`
`18
`
`ABB Inc. Exhibit 1001, Page 6 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 3 of 7
`
`US 8,095,237 B2
`
`Camera to Tool
`~~ - " Calibration
`~
`
`/
`
`Camera
`lb
`
`" ➔
`
`Tool /If
`
`Object Frame defined for each
`position in Tool Space
`
`Fig. 3
`
`~tool path memorized
`/ / at training in object frame
`
`/ ·- :_L-------~,
`//~ ✓ -~
`//
`J
`,/ ~·~-. £~-~
`
`/
`
`Object Frame 3lf
`
`4D
`
`Object Space 3 b
`
`ABB Inc. Exhibit 1001, Page 7 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`eVtSionF'octory - Demo
`fjfe !,lode f.pgin
`
`D I ~ I B I 1!S I □Tlfill I l!lil ~ I l2l
`Sou~e;:J
`[ ~;:
`c-
`
`□§I><
`l:lelp
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~
`
`? ....
`~o
`N
`0 ....
`
`N
`
`~ CJ.,--__ -=::::,..,---i-______ ~
`
`llfflG(JeS:
`fil00001 Origin: -6♦.24, 2.19, 699.08, XAxis: -63.32, -197.81, 699.133, YAxis: 35,75, 3.65, 697.17
`T:-156.56, -82.52, 85UO. R: -0.76, o:35, -G.20
`Deviations T: -0.04, -0.01, 0.00. R: 0.00, -0.00, -0.01
`
`2
`
`..__........
`
`50
`
`FIG. 4
`
`('D
`('D
`
`.i;...
`
`rJJ =(cid:173)
`.....
`0 ....
`
`-....J
`
`d r.,;_
`00 = \0
`'N w
`--..l = N
`
`UI
`
`Mehor
`Anchor2
`Pld
`1'1,11
`!'112
`l'MJ
`Pt.14
`Pl,15
`PMS
`?1,17
`PM8
`Pl.!$
`P.110
`PMII
`Plol12
`?Mil
`PMI+
`Pl,115
`
`®10900
`t@:::::J
`Q crmbor
`!Update 1m.1
`TIil ~ j Clear I
`
`S.llinga f o..iation Umltsf 1/0
`I •I
`Colil>ra\ion I M<ICol--30
`Object hm e - - - - - - - - - ,
`ObJect Aof Heighl (mm); I 1:!5.00000J
`(ia D!Sjlley
`X
`Y
`Z
`Origin (mm)!§:](§[] ~
`~ (mm}!2&3.00 I~~
`YAxls (mm )~~§ : : ]
`
`I Save gold•~ calb
`J I l.oadlmll!J•
`
`J
`
`S<M! lma<J•
`
`!Version: 1.41 Product: 0021! !Operoto,
`
`ABB Inc. Exhibit 1001, Page 8 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 5 of 7
`
`US 8,095,237 B2
`
`Calibration of the camera mounted
`on the robot arm
`
`Position the camera on the robot arm so it is orthogonal to the
`"Calibration Model". Define the "Training Space" for the robot aligned
`with the template used for calibration
`
`'
`Compute camera intrinsic parameters and the "Camera Space->
`Training Space" transformation
`
`11
`
`Compute "Camera Space-::> Tool Space"
`transformation using the "Camera Space->
`Training Space" transformation and inquiring
`the robot about the "Tool" position in "Training
`Space"
`
`FIG. 5
`
`ABB Inc. Exhibit 1001, Page 9 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 6 of 7
`
`US 8,095,237 B2
`
`Teaching The Object Features & Robot Operation
`Path
`
`Place the object in the "Training Space" and capture an image
`with the robot in the calibration position (where the "Camera
`Space-> Training Space" transformation was calculated)
`
`"
`Select from the image at least 5 visible
`features
`
`1•
`
`I Calculate feature 3D position in "Training Space"
`
`I
`
`G. 6
`Fl
`
`1
`
`Define an "Object Space " aligned with the 'Training
`Space" but connected to the object and transform
`the 30 coordinates of the features in that space
`
`Compute the "Object Space -> Camera Space" transformation using the 30
`position of the features inside this space and the position in the image by computing
`an extrinsic calibration using the camera calibration from previous step
`
`•
`Define an "Object Frame" inside "Object Space "to be used for teaching the
`robot operation path. Compute this Frame position & orientation in "Tool Space"
`using the transformation from "Object Space-> Camera Space" and "Camera
`Space-> Tool Space"
`
`1
`
`Send the "Object Frame" to the robot and train the
`robot operation path inside this space
`
`ABB Inc. Exhibit 1001, Page 10 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`U.S. Patent
`
`Jan.10,2012
`
`Sheet 7 of 7
`
`US 8,095,237 B2
`
`Object Positioning and Robot Guidance
`
`Position the robot in a predefined position above the bin
`with object.If no object is in the field of view move robot
`until an anchor feature is found in the image
`
`•
`
`Use the position and orientation of anchor feature to
`compute the expected position of the rest of the
`features; Find the position of all the visible features in the
`image;
`
`1,
`With the positions· of features from the image and their
`corresponding positions in "Object Space" (calculated in the
`training session) use the camera calibration to compute the
`transformation between the "Object Space " and"Camera
`Space". Use camera extrinsic calibration.
`
`,
`Use the transformation from above to calculate the
`movement of the robot to position the camera so that it
`appears orthogonal to the object- same position.as in
`training. In this way all the features will be as similar as
`possible to those at training This will make the recognition
`and positioning more accurate.
`
`'
`Find "Object Space-> Camera Space" transformation the
`same way as in the previous step (using the features
`positions); Compute object frame memorized at training
`using the found transformation and "Camera Space-> Tool
`Space" transformation
`
`l •
`
`Send the computed "Object Frame" to the robot;
`Use the "Tool" position to define the frame in "Robot
`Space"; Perform the trained operatin path on the object
`inside this space
`
`FIG. 7
`
`ABB Inc. Exhibit 1001, Page 11 of 19
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`US 8,095,237 B2
`
`1
`METHOD AND APPARATUS FOR SINGLE
`IMAGE 3D VISION GUIDED ROBOTICS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of U.S. patent
`application Ser. No. 10/153,680, filed May 24, 2002 now U.S.
`Pat. No. 6,816,755 which is pending.
`
`TECHNICAL FIELD
`
`The invention relates to the field of vision guided robotics,
`and more particularly to a method and apparatus for single
`image three dimensional vision guided robotics.
`
`BACKGROUND
`
`10
`
`2
`least 5 visible object features from the image; iii) creating a
`3D model of the object ("Object Model") by calculating the
`3D position of each feature relative to a coordinate system
`rigid to the object ("Object Space"); ( c) training a robot
`5 operation path by: (i) computing the "Object Space-Sensor
`Array Space" transformation using the "Object Model" and
`the positions of the features in the image; (ii) computing the
`"Object Space" position and orientation in "Robot Frame"
`using the transformation from "Object Space-Sensor Array
`Space" and "Robot-Eye Calibration"; (iii) coordinating the
`desired robot operation path with the "Object Space"; (d)
`carrying out object location and robot guidance by: (i) acquir(cid:173)
`ing and forming an image of the object using the sensor array,
`searching for and finding said at least 5 trained features; ii)
`with the positions of features in the image and the correspond-
`15 ing "Object Model" as determined in the training step, com(cid:173)
`puting the object location as the transformation between the
`"Object Space" and the "Sensor Array" and the transforma(cid:173)
`tion between the "Object Space" and "Robot Frame"; (iii)
`communicating said computed object location to the robot
`20 and modifying robot path points according to said computed
`object location.
`The invention further provides a system for carrying out the
`foregoing method.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`In drawings which illustrate a preferred embodiment of the
`invention:
`FIG. 1 is a perspective view of a vision-guided robot;
`FIG. 2 is a schematic diagram illustrating the relative
`frames of reference for calculating the position of the object;
`FIG. 3 is a schematic diagram illustrating the calculation of
`the intended operation path on the object;
`FIG. 4 is a representation of the operator's computer screen
`for selecting reference features on the object;
`FIG. 5 is a flow chart illustrating the calibration of the
`camera mounted on the robot arm;
`FIG. 6 is a flow chart illustrating the method of teaching the
`object features and handling path; and
`FIG. 7 is a flow chart illustrating the method of object
`positioning and handling.
`
`DESCRIPTION
`
`Robots have long been widely used in manufacturing pro(cid:173)
`cesses for many applications. Many different types of sensors
`are used to guide robots but machine vision is increasingly
`being used to guide robots in their tasks. Typically such
`machine visio