Visual Information Assist System
`Using 3D SOKUIKI Sensor for Blind People
`– System Concept and Object Detecting Experiments –
`
`Tatsuro UEDA1, Hirohiko KAWATA1, Tetsuo TOMIZAWA2,
`Akihisa OHYA1 and Shin’ich YUTA1
`
`1Inteligent Robot Laboratory, University of Tsukuba
`{t_ueda,kwt,ohya,yuta}@roboken.esys.tsukuba.ac.jp
`http://www.roboken.esys.tsukuba.ac.jp
`2National Institute of Advanced Industrial Science and Technology (AIST)
`t.tomizawa@aist.go.jp
`
`Abstract— The purpose of this research is to develop a system which
`gives blind people information of the environment around them. A per-
`son is equipped with a 3D scanner and a small sized PC while walking. The
`scanner scans and acquires 3D range data map of the environment. The
`PC analizes the range data map and detects objects which are useful for
`blind people in orde to walk. The PC gives environmental information to
`them by synthesized sound. This paper first introduces the concept of the
`whole system and clarify the tasks for realizing the system. Secondly the
`method for acquisition of 3D range data and detecting objects and obsta-
`cles are described. Then the usefulness of our proposed system is examined
`by an experiment in which our trial system detects bumps and trenches in
`the experimental environment.
`
`I. INTRODUCTION
`
`In recent years accessibility of everyday-life environments for
`disabled and aged people attracts public interest. Actually there
`are a lot of improvements for it, such as textured paving blocks,
`slopes instead of steps, handrails, elevators, etc. The movement
`of universal design proposed by Ronald L. Mace has been more
`and more prosperous. However, these improvements are limited
`to specific places and it is still hard for the disabled to live in
`most of places at present. Especially for blind people who have
`no visual information, there are a lot of dangers in everyday-life
`environments. These people have a lot of difficulties to acquire
`environemental information. Moreover, obstacles which are not
`dangerous to ordinary people are able to become dangerous to
`them. Though they use blind stick to acquire these information,
`it is still hard for them to walk around in most of the places.
`A lot of studies have been done to develop a system which
`assists blind people and these studies are roughly classified into
`two groups. In one group, a blind person walks with a guide
`robot and the robot guides him to a destination avoiding obsta-
`cles [1], [2]. However these guide robots limit the person’s field
`of activities, because the robots have difficulty in walking on a
`crowded street or going through irregular ground. Another ap-
`proach is to develop an intelligent blind stick which has sensors
`to help finding objects [3]. With this kind of blind stick, a blind
`person is able to find objects even if the stick does not touch
`them. The weak point of these intelligent sticks is that they can
`not cover the whole environment around the person.
`
`On the basis of these problems above, we propose a visual
`information assist system using a 3D scanner. In this system, a
`blind person is equipped with a 3D scanner and a small sized
`PC. While walking, the 3D scanner scans the environment in
`front of the user and then the PC analyzes environmental data to
`acquire these information. The PC tells the information of envi-
`ronment to the person by synthesized sound. Using this system,
`blind people will be able to obtain visual information of the en-
`vironment around them by sound.
`This paper first describes the concept of the whole system
`(section II). Secondly, the method for constructing 3D range
`data map and detecting objects are described (section III, IV).
`Potential usefulness of our proposed system is examined by an
`experiment in which proposed system which is experimentally
`implemented is fastened to the environment, constructs environ-
`mental data map and detect obstacles (section V, VI).
`
`II. CONCEPT OF THE SYSTEM
`Images of the concept of our proposed system are shown in
`Fig.1 (a) and (b). As shown in Fig. 1 (a), a person is equipped
`with a 3D scanner on his chest and has a small sized PC in his
`bag. The scanner and the PC are connected to each other. The
`scanner scans the environment in front of the person and then
`the PC analyzes the 3D range data continuously. In data analyz-
`ing, the PC first constructs 3D environmental map and then de-
`tects objects and obstacles which seem to be useful to the blind
`person, such as steps, doors, walls, bumps etc. As soon as any
`objects or obstacles are found, the PC tells the user these infor-
`mation by synthesized sound (see Fig.1 (b)). The system gives
`them the relative position, configuration and the name of the
`objects or obstacles so that they can easily recognize what and
`where it is located in the environment.
`Considering that a human generally walks 1m in a second
`and that it takes some time to give these kind of information by
`sound, the maximum scanning range of the 3D scanner should
`be 5m at least, otherwise, the PC might tell him information of
`objects or obstacles, after they have passed these objects or col-
`lided with these obstacles. The function of the system is roughly
`
`1-4244-0136-4/06/$20.00 '2006 IEEE
`
`3058
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`Exhibit 1024 page 1 of 6
`DENTAL IMAGING
`
`

`

`classified into two parts shown below.
`1. Acquisition of 3D environmental data.
`2. Detecting objects and obstacles from the 3D environmen-
`tal data.
`The basic methods for these two subfunctions are described
`in the next two sections.
`
`the appearance of the sensor and tableI shows the main specifi-
`cations of it. This LRF is light, compact and accurate, so it does
`not place any significant burden on the person equipped with the
`scanner. We use this LRF for our 3D scanner.
`
`Side
`
`Front
`
`(a)
`
`steps,
`11 o'clock
`2
`
`· ·
`
`(b)
`
`Fig. 1. Concept of the system : (a) A person equipped with a 3D scanner and a
`PC, (b) A person walking with the system
`
`III. ACQUISITION OF 3D ENVIRONMENTAL DATA
`This section describes the method for acquisition of 3D envi-
`ronmental data.
`
`A. Consideration of the Sensor
`As it is described in the previous section, the information
`which is given to a blind person is position, configuration, names
`of objects and obstacles. To get such information, range data of
`the environment is important. In acquisition of 3D environmen-
`tal data, cameras and laser range finders are commonly used.
`Using cameras for getting 3D information has some advantages.
`There is a lot of information in a camera image and it requires
`little time to be taken. However camera images are affected by
`light condition and also it has difficulty to capture 3D configu-
`ration data of the environment, so it is not appropriate for our
`system. On the other hand, laser range finders can get accurate
`range data and it is not affected by light condition. Therefore we
`use laser range finder for our 3D scanner.
`A SOKUIKI sensor URG-04LX[4], which is a super small
`sized 2D Laser Range Finder (LRF) made by HOKUYO AU-
`TOMATIC, is the most compact commercial LRF. Fig.2 shows
`
`Fig. 2. SOKUIKI sensor URG-04LX
`
`Detectable distance
`Accuracy
`
`Resolution
`Scanning angle
`Angle resolution
`Scanning time
`Weight
`External dimension
`
`0.02 to 4[m]
`0.02 to 1[m]:±10[mm]
`1 to 4[m]: ±1% of distance
`1[mm]
`240[degrees]
`Apporox. 0.36[degrees]
`100[ms]
`Apporox. 160[g]
`50×50×70[mm]
`
`TABLE I
`SPECIFICATION OF URG-04LX
`
`B. Method for Acquisition of Data Set for 3D Range Data map
`A 2D LRF measures only 2D range data. Changing sensor’s
`posture is a common technique to make 3D scanning feasible.
`There are different ways to change its posture[6]. Our rotating
`method, shown in Fig.3(a), has horizontal scan plane and rotates
`its plane around x-axis. With this method, measurement points
`focuses on x-axis (see Fig.3(b)). To represent 2D range data
`in Cartesian coordinates, the position and posture of 2D LRF
`is needed. The angle of the sensor is determined by counting
`pulses from motor encoder which rotates the sensor.
`
`Rotation Axis
`(X-Axis)
`
`(b)
`(a)
`Fig. 3. Scanning Method : (a) Rotating Method, (b) Measurement Density
`Distribution
`
`As shown in tableI, it takes 100ms for the sensor to make a 2D
`plane scan. It means that acquisition of enough 3D range data
`for recognizing environment takes some seconds. The 3D scan-
`ner is attached to a blind person’s body, so the scanner changes
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`3059
`
`Exhibit 1024 page 2 of 6
`DENTAL IMAGING
`
`

`

`its position and posture dynamically, the position and posture
`of the scanner also needs to be estimated for constructing 3D
`range data map. We use an inertial measurement unit (IMU),
`which consists of three gyroscopes and three accelerometer, for
`estimation of scanner’s position and posture. 3D range data map
`is calculated using LRF sensor, angle estimation module and gy-
`roscopes and acceleration sensors as shown in Fig.4.
`
`{it-n J / n 11-n )
`I
`I
`
`I $1)1..'t"lli:I I rJlftCl'd•O
`
`,1:11lJ.•I
`
`lWll.k .,-w.;wllti1111
`11'111•1111~
`
`""tl,,11r
`8C>K I. 11,.1 >em,m
`
`I
`
`JDu1w,:
`dd•1t1:.p
`
`1::~rll &
`x,..,,_t,;n,IWM ....:,u-w
`
`I
`
`<If' t.l'.onn<'r
`
`I 1111-.iliYJJ llll•J ,.~1111111:
`
`I
`
`Fig. 4. Structure of the 3D scanner
`
`C. Timing Consistency
`
`As it was described already that the position and posture of
`the scanner and the rotating angle of the sensor at the moment
`when each measurement point is measured, is needed to con-
`struct 3D range data map. However, these devices are not syn-
`chronized with each other, which makes it difficult to capture
`the position, the posture and the angle. The SOKUIKI sensor
`URG-04LX transmits a synchronous signal right after it finishes
`scanning. Also the range data of the sensor has accurate time
`stamp. We have already developped the method to consistent
`data of the SOKUIKI sensor and other data [9]. Synchronous
`signals consistent range data and angles of the sensor and time
`stamps do range data and position and posture of the scanner.
`
`IV. RECOGNITION OF ENVIRONMENTAL INFORMATION
`
`This section describes preparation of 3D range data map and
`the method for detecting objects and obstacles. A 3D range data
`is constructed and divided by voxels. Then objects and obstacles
`are detected by using template matching algorithm.
`
`A. Preparation of 3D Range Data Map
`
`Environment object information, especially information of
`obstacles, should be informed to the person as soon as possi-
`ble. For this reason, 3D range data map should be constructed
`each time the LRF scans the environment. The purpose of our
`reaserch is to give the information of the environment around a
`blind person, so the map does not need to include all the range
`data which has been scanned. When the range data scanned in
`α seconds are enough to construct 3D range data map, the latest
`range data scanned in α second are used to construct the map
`after every scan finishes (Fig.5).
`Though the range data map is constructed with millimeter ac-
`curacy, we can not recognize millimeter difference. Therefore
`3D range data map is divided by voxels of 50×50×50[mm].
`The size of each voxel is determined on the basis of the accu-
`racy of the LRF. The voxels have only a single volume.
`If a
`measurement point is included in a voxel, that voxel has vol-
`ume.
`
`Fig. 5. SOKUIKI sensor URG-04LX
`
`B. Detecting Method
`We use 3D template matching algorithm to detect objects and
`obstacles. Firstly templates of the objects and obstacles, such as
`steps, doors, a bump etc. are prepared beforehand. When the
`voxel map is constructed, the map is scanned by each template.
`If the correlation between a template and an area of the map is
`lower than threshold, an object locates in this area. The correla-
`tion C(u,v,w) is evaluated by the equation shown below in the
`standard 3D template matching algorithm.
`
`C(u,v,w) =
`
`d∑
`
`h∑
`
`w∑
`
`z=0
`
`y=0
`
`x=0
`
`|T (x,y,z)− M(u + x,v + y,w + z)|
`
`(1)
`
`Variables u, v and w are represent a position of a voxel in the
`map and x, y, z represent that of the template. Constant numbers
`d, h and w each represents depth, height and width of a template.
`The problem of 3D template matching algorithm is the cost
`of processing. In 3D space, objects have 6 degrees of freedom,
`3 for its position and the others for its posture. The equation
`shown above covers only position. It means that we prepare a
`number of templates for all postures and the calculation pro-
`cesses for each template. Though objects have 6 degrees of
`freedom, each object has different constraints. For example, the
`door is attached to the wall and vertical to the ground. The steps
`are along walls and on the ground. These constraint conditions
`decrease the cost of processing drastically. Specific method for
`detecting objects and obstacles depend on them.
`
`V. EXPERIMENTAL IMPLEMENTATION
`This section describes an experimental implementation which
`is setup in order to examine potential effectiveness of our pro-
`posed system. As it was stated in introduction, the usage of this
`experimental system is to be fastened in the environment and de-
`tect bumps and trenches which are common obstacles for blind
`people
`
`A. Rotating Device and Angle Estimation Module
`A rotating device is shown in Fig.6(a). A DC motor manufac-
`tured by Maxon Motor rotates the sensor. A slip ring manufac-
`tured by TSUBAME Radio Co.,Ltd is attached to the rotating
`shaft and all the cables from the sensor are through it so that
`the sensor rotates unlimitedly (see Fig.6(b)). Note that the max-
`imum rotating velocity of the slip ring is 60 rpm. The angle of
`the sensor is estimated by an angle estimation module which we
`have developed in our previous research. The module estimates
`the angle of the sensor every 5 ms by counting pulses from the
`motor encoder. The module also receives synchronous signals
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`3060
`
`Exhibit 1024 page 3 of 6
`DENTAL IMAGING
`
`

`

`from the sensor and put a time stamp to the angle data. This time
`stamp makes consistent of range data and angle data possible.
`
`6 1
`
`-
`
`8
`
`(b)
`(a)
`Fig. 6. Rotating device : (a) overall view, (b) slip ring
`
`B. Rotating Velocity
`In constructional 3D scanning methods with a SOKUIKI sen-
`sor, the scan plane of the sensor is usually rotated slowly at 10
`rpm or less. The increase of the measurement points with this
`method is shown in Fig.7(a) and the view from x-axis is shown
`in Fig.7(b). The numbers in Fig.7(b) represent the order of scan-
`ning. We see it from Fig.7 that the measurement points increase
`in two directions. However the 3D range data map has two big
`holes which have no measurement point until the sensor rotates
`180 degree. The scanner cannot detect objects and obstacles in
`these holes and human might bump against it.
`To solve this problem, we rotate the sensor fast, around 30
`rpm. Fig.8(a) and (b) show the increase of the measurement
`points. If we see Fig.8 we will see that the measurement points
`increase from many directions. This rotating method disperses
`holes in the range map. When rotating the sensor fast, it should
`be noted that the sensor scans at a same angles after certain pe-
`riod of time, depending on rotation velocity. The rotation veloc-
`ity should be determined carefully.
`
`I
`
`9
`
`/2
`< 1/
`
`10
`
`I <•
`(b)
`(a)
`Fig. 8. Change of measurement points with the sensor rotating fast: (a) Bird
`view, (b) View from x-axis
`
`of trenches. Fig.10 shows only 4 templates. The point symme-
`tries of these are the others. Detecting area of trenches are on
`the ground.
`
`■ : "oxel with "olume
`0 : ,•oxcl wilh n o volume
`
`Fig. 9. Template for detecting bumps
`
`■ : voxel with volum e
`D : vuxcl w ith nu volume
`Fig. 10. Parts of templates for detecting trenches
`
`(a)
`
`(b)
`Fig. 7. Change of measurement points with the sensor rotating slowly : (a) Bird
`view, (b) View from x-axis
`
`C. Templates for Detecting Bumps and Trenches
`The purpose of the experimental system is to detect bumps
`and trenches around a person. Fig.9 and Fig.10 each shows a
`template for a bump and parts of template for trenches. The
`template for bumps consists of 3 voxels which are attached to
`each other vertically. The detecting area for bumps are above
`the ground. 8 templates which are combination of a voxel with
`volume and 2 voxels without voxels are used for detecting edges
`
`VI. SCANNING AND DETECTING EXPERIMENT
`An experiment is performed to evaluate potential of our pro-
`posed system. The environment for the experiment is shown
`in Fig.11(a). The 3D SOKUIKI sensor is fastened in the envi-
`ronment at a height of 1150 mm and scans environment. The
`velocity of rotation is 30.21 rpm. Fig.11(a), (b) and (c) show 3D
`range data map scanned in 2s, in 5s and in 10s. Measurement
`points which are higher than 2 m are not displayed to facilitate
`clear view of the data map. Fig.11(b), (c) and (d) shows that the
`measurement points increases relatively uniformly. Though the
`sensor rotates fairly fast, the scanning results are accurate.
`Fig.12 shows detecting results. The green points in Fig.12
`(a) represents bumps in the environment. We see from Fig.12
`(a) that a lot of green points are found along the walls, in front
`of steps and in the middle of steps. The blue points in Fig.12
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`3061
`
`Exhibit 1024 page 4 of 6
`DENTAL IMAGING
`
`

`

`(b) which is viewed from the top show edges of trenches. The
`results show that the experimental system is able to detect bumps
`and trenches.
`
`VII. CONCLUSION
`In this paper, we proposed and described a visual assist sys-
`tem for blind people. The system which is implemented ex-
`perimentally showed the effectiveness in constructing 3D range
`data map and detecting obstacles in the experiments. The exper-
`imental system cannot estimate its own position and posture at
`present. An IMU is going to be implemented for calculating its
`position and posture. The experiment showed that bumps and
`trenches were localized and recognized. The next step on this
`research is to detect walls, stairs and many other objects which
`are found in common environment.
`
`VIII. ACKNOWLEDGEMENT
`We are grateful to Mr. Luis Yoichi Morales Saiki of the
`University of Tsukuba for his assistance in preparing this
`manuscript.
`
`REFERENCES
`[1] Lacey, Gerrard and Kenneth Dawson-Howe : “Evaluation of Robot Mobil-
`ity Aid for the Elderly Blin” Proceedings of the Fifth International Sympo-
`sium on Intelligent Robotic Systems, Stockholm Sweden 1997
`[2] Kentaro Iwatsuka, Kazuhiko Yamamoto, Kunihito Kato : “Development
`of a Guide Dog System for the Blind with Character Recognition Ability”
`crv, pp. 401-405, 1st Canadian Conference on Computer and Robot Vision
`(CRV’04), 2004.
`[3] Sung Jae Kang, Young Ho-Kim, In Hyuk Moon : “Development of an Intel-
`ligent Guide-Stick for the Blind” Proceeding of the 2001 IEEE International
`Conference of Robotics & Automation, Seoul Korea 2001
`[4] Hirohiko Kawata, Toshihiro Mori, and Shin’ichi Yuta : “ Design and Real-
`ization of 2-Dimensional Optical Range Sensor for Environment Recognition
`in Mobile Robots”, Jounal of Robotics and Mechatronics pp.116-120
`[5] Majd Alwan, Matthew B Wagner, Glenn Wasson, and Pradip Sheth : “ Char-
`acterizaton of Infrared Range-Finder PBS-03JN for 2-D Mapping”, Proceed-
`ings 2005 IEEE Int. Conf On Robotics and Automation
`[6] Oliver Wylf and Bernardo Wagner : “Fast 3D Scanning Methods for Laser-
`Measurement Systems”, Proceedings 2003 International Conference on Con-
`trol Systems and Computer Science
`[7] Albert Diosi and Lindsay Kleeman : “Laser Scan Matching in Polar Cood-
`inates with Application to SLAM” , Proceedings 2005 IEEE/RSJ Interna-
`tional Conference on Intelligent Robots and Systems
`[8] Kazunori Ohno, Takashi Tsubouchi and Sshin’ich Yuta :“Outdoor Map
`Building Based on Odometry and RTK-GPS Positioning Fusion” Proceed-
`ings of the 2004 IEEE International Conference on Robotics and Automation
`[9] Tatsuro Ueda, Hirohiko Kawata, Tetsuo Tomizawa Akihisa Ohya and
`Sshin’ich Yuta :“Mobile SOKUIKI Sensor System - Accurate Range Data
`Mapping System with Sensor Motion” Proceedings of the 2006 International
`Conference on Autonomous Robots and Agents (to be appeared)
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`3062
`
`Exhibit 1024 page 5 of 6
`DENTAL IMAGING
`
`

`

`""
`
`,I, , f .~
`
`,11,/'
`
`,
`
`r
`
`,
`
`rf/t ,
`
`(a)
`
`(b)
`
`, ..
`
`"'
`
`(d)
`(c)
`Fig. 11. 3D range data map : (a) Experimental environment, (b) Scanning in 2 sec, (c) Scanning in 5 sec, (d) Scanning in 10 sec
`
`(a)
`(b)
`Fig. 12. Detecting obstacles : (a) Result of detecting bumps, (b) Result of detecting trenches
`
`Authorized licensed use limited to: Foley & Lardner LLP. Downloaded on May 08,2022 at 22:28:26 UTC from IEEE Xplore. Restrictions apply.
`
`3063
`
`Exhibit 1024 page 6 of 6
`DENTAL IMAGING
`
`