Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person

Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person Hindawi Applied Bionics and Biomechanics Volume 2020, Article ID 6153128, 11 pages https://doi.org/10.1155/2020/6153128 Research Article Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person 1 1 2 1 1 Shahid Akram, Ali Mahmood, Ihsan Ullah, Muhammad Tahir Mujtabah, Ali Bin Yasin, 1 3 1 Asif Raza Butt, Muhammad Shafique, and Sajjad Manzoor Department of Electrical Engineering, Mirpur University of Science and Technology (MUST), Mirpur, 10250 AJK, Pakistan Department of Electrical Engineering CUI, Abbottabad Campus, Abbottabad, KPK, Pakistan Department of Biomedical Engineering, Riphah International University, Islamabad, Pakistan Correspondence should be addressed to Sajjad Manzoor; sajjad.ee@must.edu.pk Received 26 December 2019; Revised 9 July 2020; Accepted 19 September 2020; Published 15 October 2020 Academic Editor: Mohammad Rahimi-Gorji Copyright © 2020 Shahid Akram et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. In this paper, we have given the design and development of a new wearable device that assists visually impaired individuals to travel independently and confidently. The newly proposed device is based on range-based sensors and would work effectively in both indoor and outdoor conditions. It is constructed in the form of two separate modules; one module is designed such that it can be attached to the waist belt of the user, and the other module is designed to wear it on ankle of the user. Both the modules communicate with each other using wireless communication and can cover the full front environment of the user. The information about the front environment is transmitted to the user, via headphone, by sending a set of voice instructions, stored in a memory card added in the belt module. In order to use the device in crowd mode, appropriate networking techniques were also implemented in the prototype such that the interference of two or more devices in the close vicinity can be avoided. In the end, effectiveness of the device is analyzed and proved by conducting experiments and obtaining statistical results. [4–6]. In Figure 1, different assistive devices for the blind 1. Introduction are shown, which can be helpful during the motion of the With the advances in technology, commercial hardware and user. The ETAs use sensors to detect obstacles in front of software applications are developed to make life easy for peo- their users and give information about the front environment ple with physical weakness. At least 2.2 billion people of the and guide the user in a manner that they would safely move forward. In some other devices, cameras are used for world have a vision impairment or blindness [1]. They may belong to a category, with moderately visually impairment, vision-based assistance of their user during motion. In some severely visually impairment, or totally blind. Most of the cases, Raspberry Pi cameras [7] are used to detect objects in visually impaired people move by using conventional front, and in other cases, Kinect sensors [8] are used to calcu- methods, i.e., white canes, guide dogs, tactile paving, and in late the distance of the user from the obstacles. Smartphone some other cases move with the help of another person, camera-based elevator finder application was developed in called as sight guide [2, 3]. With the evolution of technology, [9]. This would help users to find if they are moving in the there is an immense need for the development of easy-to-use direction of the elevator or not. Mobile phone camera, with devices that would be helpful for the visually blind people in color codes was used in [10] in order to convey information mobility. In this way, visually impaired individuals may about the inside of a building. In [11], the authors have used travel independently with confidence and participate in daily fusion of artificial vision and GPS for locomotion assistance activities. and obstacle detection for the user. Fusion of ultrasonic sen- Recently, many types of electronics travel aids (ETA) sor, GPS, and GSM was introduced in a silicon glove in [12]. have been developed for mobility assistance of blind people A walking stick that uses radiofrequency identification (RFI) 2 Applied Bionics and Biomechanics (a) (b) (c) (d) Figure 1: Different mobility assistive devices for visually impaired people: (a) [18], (b) [11], (c) [15], and (d) [7]. to solve all the issues, mentioned above, in this paper, we was constructed in [13] in order to help blind people navigate on their sidewalk. However, most of the assistive device for have proposed an assistive device for the blind which is inex- blind uses range-based sensors, which are cost effective, have pensive and lighter in weight. It is a wearable modular device. good availability, and easy to operate. These user friendly It is equipped with range-based sensors. These modules can devices use IR sensors [14], for small range; ultrasonic sen- be attached to the waist belt and the ankle of the user. This device covers more area in front of the user and is effectively sors [15, 16], for medium range; and/or LIDAR [17], for long range, to detect and localize obstacles in front of the user. helpful for his mobility. The experimental analysis on the Most of the assistive devices for the blind, developed, till device is also done to check its effectiveness. now, are either very costly or bulky for the user to wear or This paper is further organized as follows: in Section 2, hold. The conventional white cane is already heavy and addi- the purpose for the design and development, along with con- tion of any device on it can make it more difficult to hold. siderations taken during design phase of the new device for Vision-based devices, with cameras attached to them, mostly visually impaired person, are given. The architecture as well belong to the categories that are both heavy and expensive. as the components used in the device is also discussed in this These days, small cameras are available; however, cost section. The algorithm designed for working of the device is remains the matter of concern in them. Similarly, smart- given in Section 3. The model for the state estimator is dis- phones, which are used in some applications for assistance cussed in Section 3.1. In Section 3.2, we have considered for visually impaired, are also expensive. The devices based the issue of communication within the device in the crowd on ultrasonic sensor or any other range-based sensors do mode, i.e., crowded environment with multiple users at same not cover all the features of the front environment. In order place. In Section 4, the experimental and statistical results of Applied Bionics and Biomechanics 3 The belt module (BM) is designed in such a way that it the device are given, so that the effectiveness of the device for visually impaired people can be evaluated. Finally, in Section can be attached to the waist belt of the user. The architecture 5, the conclusion of the paper is given. The expected future of the belt module is shown in Figure 3. It serves as the master development in the proposed device is also discussed in this module for the ankle module (AM), and it has its own section. rechargeable power supply. This module has three ultrasonic sensors targeted in different directions. The first sensor is directed normal to the user body, and it would detect any 2. An Assistive Device for Visually Impaired obstacles in front of the user. The second sensor is directed People (aVIP) in upward direction at an angle of 45 from first sensor. This sensor can detect obstacles only at head level such as lower Design, construction, and development of a new assistive roof or lower inclined side of the stairs. A third sensor is device for visually impaired people are given in this section. adjusted in such a way that it is directed downward at an One of the intentions of the research is to develop a cost- angle of -45 from the first sensor. It can detect any pits or effective assistive device for visually impaired people. hole in the ground in front of the user. It would also detect any obstacle of small size or step, not detected by the first sen- 2.1. Design Consideration for Assistive Device. During the sor. However, due to its orientation, the sensor would only be design phase of the device, the following considerations were able to detect small obstacles such as step or footpath unless made to get maximum information about the front the user reaches very close to it. In combination with the environment: ankle module this sensor would also detect small, hollow obstacles, i.e., tables and chairs. The developed belt module (1) The device should determine all obstacles in front of is given in Figure 4(a). the user body, from the ground to the head, as shown The belt module has two, commercially available, Ardu- in Figure 2(a). It should be able to distinguish obsta- ino Mini microprocessors, connected with each other. This cles at the following locations: is due to the reason that each Arduino Mini has only one SPI interface while the belt module needs two SPI interfaces, (i) Large obstacles in front of the user, as shown in i.e., one for a SD card that stores audio instructions and other Figure 2(b) for a communication device. The memory card is connected (ii) Holes and pits, on the ground, in front of the to one of the microprocessors. This memory card would have user, as shown in Figure 2(c) audio instructions recorded in it, each as a separate audio file. A headphone is also connected to the device through an (iii) Hanging or inclined obstacles at the level of the audio amplifier. An nRf24L01 module acting as a receiver is head of the user, as shown in Figure 2(d) connected to the other microprocessor in the belt module, (iv) Small object or stairs in front of the user, as so that the information from the ankle module can be wire- shown in Figure 2(e) lessly received. The architecture of the ankle module is also shown in (v) Hollow objects (i.e., tables and chairs) in front of Figure 3. It has a single Arduino Mini microprocessor con- the user nected to a rechargeable battery. It has one ultrasonic sensor in it which is directed in the direction perpendicular to the (2) Information about the environment, in front, should user body. This sensor can detect all the obstacles of small be given in the form of a clear voice instructions. size, which are undetectable to the sensors in the belt module. (3) The instructions should be easy to understand; this An inertial measurement unit (IMU) sensor is attached to the could be recorded in the user’s own voice. module to detect the dynamics of leg motion and differenti- ate straight and bent leg during swing or stance phase of In Figure 2(a), the layout of the proposed device is given motion. Using the IMU sensor, only obstacles in front of that would accomplish all of the abovementioned tasks. A the user would be detected and the ground would not be con- modular device, consisting of two modules with four differ- sidered as an obstacle when the leg would be in bent position. ent proximity sensors, is proposed. More detail of the archi- It can also be used to count the number of steps taken by user. tecture of the proposed device is discussed in Section 2.2. The data from the ankle module is transmitted to the belt module through the nRF24L01 module. The nRF24L01 mod- 2.2. Architecture of aVIP Device. In order to cover the maxi- ule in the belt module acts as a receiver while the one in the mum area in front of the user, the device is designed in a ankle module acts as a transmitter. The constructed ankle modular form by combining sensors, commercially available module is given in Figure 4(b)). microprocessors, transmitter, and receiver. The architecture of the device is given in Figure 3. It is subdivided into two 3. Working of the Device and Algorithm for modules that communicate with each other without any wire. Obstacle Detection These modules are named as There are three ultrasonic sensors installed at waist height to (i) Belt module measure the distance to the ground or an obstacle. One is (ii) Ankle module installed normal to the body of user, the other up waist at 4 Applied Bionics and Biomechanics For head Waist level belt obstacles module For large obstacles in front For pits in ground in front Ankle module For small obstacles in front (a) (b) (c) (d) (e) Figure 2: Evaluation for design consideration of an assistive device for visually impaired people. (a) Layout of the device. (b) Obstacle at head level. (c) A pit hole on a straight path. (d) Obstacle upfront. (e) Stairs or small obstacle in front. ° ° All the sensors perform differently under different ter- 45 , and the third down waist at an angle of negative 45 -down to normal. The device installed on the ankle has an ultrasonic rains and scenarios. Some of the common scenarios that a sensor directed normal to the body as well as an IMU. The belt subject may face have been evaluated and considered in the and ankle modules detect the obstacle separately. However, design of the hardware, as shown in Figures 2(b)–2(e). The only the belt module has an SD card connected to it that con- outputs from all the sensors are used to warn the user about tains audio instructions for the user about the nature of the any obstacle ahead. The first sensor to register this kind of obstacles in front. Furthermore, headphones are also con- obstacle will be the normal sensor. In case of a hole on a nected to belt module. Therefore, the belt module acts as the leveled surface, the primary action or warning is given to master and the ankle module acts as a slave. the user based on the input from the waist down sensor; Applied Bionics and Biomechanics 5 Initial Anckle module setting Ultrasound Ultrasound SD – memory sensor sensor card Arduino Mini Audio amplifier Arduino Mini Arduino IMU sensor Battery Mini Battery Headphone NRF24L01 NRF24L01 transmitter reciever Belt module Figure 3: Architecture for modular assistive device for visually impaired people. (a) (b) Figure 4: Newly constructed modular assistive device: (a) belt module and (b) ankle module. however, the ankle sensor with IMU sensor output can be a user wears a device for the first time, it should be operated used to navigate the obstacle and scan the obstacle for clear- in setting mode to adjust these limits. During the setting ance. In the scenario of the leveled surface with obstacle at mode, the user is instructed to stand on a plane surface and head level only, up waist sensors gives distance. While in hold a plane paper sheet at head level. By pressing setting the case of stairs or small obstacle ahead, the normal sensor switch, the device would automatically adjust obstacle detec- is again the first sensor to register an obstacle, but the classi- tion distance limits l ðlimitÞ and l ðlimitÞ for the head and h g fication of the obstacle can only be performed by the down ground levels. In order to avoid false detection due to uneven waist sensor and the ankle sensor. ground, small clearances ϵ , ϵ , and ϵ are added in the algo- 1 2 3 The outputs of sensors are passed through a moving aver- rithm. Once the setting is done, the setting mode is closed age filter before using them for a decision. Since the moving and the device is operated on working mode. For instanta- average filter is also a low-pass filter, it smoothens any abrupt neous values l , l , and l of the respected normal, head level, b h g transitions thus eliminating any noise or erroneous measure- ground level sensors in the belt module and values l of ankle ments. The moving average filter implemented in the hard- sensor, the algorithm for operation of the assistive device is ware is depicted in Figure 5. The assistive device has two given in Algorithm 1. modes of operation: setting mode and working mode. Same obstacle detection distance limits l ðlimitÞ and l ðlimitÞ, for b a 3.1. The State Estimation Process. Since the measurements, the front sensor in the belt and ankle modules, respectively, especially for moving obstacles, are of stochastic nature, a can be adjusted for the users of different heights. Therefore, Kalman Filter (KF) can be deployed for target state estima- there is no need to change these parameters. However, obsta- tion. The measurement from the sensor consists of range R, cle detection distance limits l ðlimitÞ and l ðlimitÞ for the h g to the moving object, i.e., z = R the time to react, τ, is not respective head and ground sensors in the belt module can estimated for a few initial scans; afterwards, it is initialized be affected by the height of each user. Therefore, whenever from the estimated range and range rate and is made part 6 Applied Bionics and Biomechanics the estimation part can be written as follows, where first Input + Output we compute the innovation covariance matrix by D χ χ 1 ξ = H Γ H + R, ð7Þ k k k/ðÞ k−1 k where R is the measurement variance and is computed from the three-sigma bound of the sensor. The Kalman gain is given by χ −1 Δ = Γ H + ξ , ð8Þ k k/ k−1 D ðÞ k k such that, the estimated state vector can now be calcu- lated from the Kalman gain and the measurement residue as Figure 5: The moving average filter used as a low-pass filter for the raw sensor input. χb = χb + Δ z − H χb : ð9Þ k k k/k k/ðÞ k−1 k k/ðÞ k−1 of the filtering process. The final state propagates in discrete Finally, the state covariance matrix is computed by the domain by following equation: χ = ϕχ + U + w , ð1Þ k k−1 k k Γ = I − Δ H Γ : ð10Þ k/k k k/ k−1 k ðÞ where χ is the state vector, U = ½00 T and the plant k k The plant covariance matrix is initialized based on the noise, w , is assumed to be white Gaussian noise with zero process explained in [20]. The quality of the estimate mean. The plant noise is characterized by a known covari- depends on the sampling time T along with the assumed ance matrix Q. The state vector consisting of the range, R, measurement process noise covariance matrix R and process range rate R, and time to react, τ,isdefined as noise variance q. _ ð2Þ χ = : R R τ 3.2. Crowd Mode Communication. The nRF24L01 chip is used for communication purpose, between two modules of The state transition matrix ϕ in Equation (1) is given as the constructed assistive device [21]. It is a radio transceiver with a frequency operating range of 2.4-2.5 GHz in the ISM 2 3 1 T 0 band. This chip was used due to its low cost, size, and low 6 7 power consumption as compared to other available options 6 7 ϕ = 01 0 , ð3Þ 4 5 [22]. Each transceiver can use 125 channels with a channel switching time less than 200 ms and data rate of up to 00 1 1 Mbps. This implies that 125 different devices can operate in the same environment without interfering with each other. with T being the sampling time. The plant noise covari- The assistive device for the blind, given in paper, was first ance matrix Q can be expressed as developed for assistance of little children at schools for blind. 2 3 4 2 Therefore, it can have more than 125 users, which can cause T T problems in communication. Thus, a remedy for limitation 6 7 4 2 6 7 in the number of communication channels is required. The 6 7 6 T 7 2 number of devices operating in an environment however Q = q: , ð4Þ 6 7 T 0 6 7 can be increased by utilizing time division multiple access 6 7 4 5 8 (TDMA) [23], which however, is not supported by the 00 nRF2401 chip. The device is made to operate in two modes, the first one being the normal mode and the second one crowd mode. The crowd mode utilizes a custom nonstandard where process noise variance is denoted by q. The KF update TDMA technique. Each device can transmit for 250 ms only and prediction equations as given in literature [19] can be whereas the receiver listens throughout and receives data expressed as from the transmitter already registered with it. The Probabil- ity density function of the uniformly distributed slot interval χb = ϕχb + U + w : ð5Þ k k k/k−1 ðÞ k−1 /ðÞ k−1 on an interval with a width (b-a) of 250 ms is given in terms of the Heaviside step function by With HsðÞ − a −HsðÞ − b b b Ps ðÞ = : ð11Þ Γ = ϕΓ ϕ + Q, ð6Þ k/k−1 ðÞ k−1 /ðÞ k−1 b − a 3 Applied Bionics and Biomechanics 7 (1) Manual setting of distance limit l ðlimitÞ in belt module for obstacle normal to the user, and distance limit l ðlimitÞ in ankle module for obstacle in front direction of the user. (2) Setting mode: If setting button is on; (a) l ðlimitÞ = current distance calculated by 45 -up sensor. (b) l ðlimitÞ= current distance calculated by 45 -down sensor. (3) Operating mode: if setting button is off; (I) If l ðlimitÞ Instruction 1: “Obstacle in front” (II) If l < l ðlimitÞ & l > l ðlimitÞ a a b b Instruction 2: “Small obstacle in front” (III) If l < l ðlimitÞ h h Instruction 3: “Obstacle at head-level” (IV) If l > l ðlimitÞ + ϵ g g 1 Instruction 4: “Pit in ground” (V) If l < l ðlimitÞ + ϵ & l < l ðlimitÞ g g 2 a a Instruction 5: “Small obstacle or footpath ahead” (VI) If l < l ðlimitÞ − ϵ & l > l ðlimitÞ g g 3 a a Instruction 6: “Table or chair ahead” Algorithm 1. Algorithm for working of the proposed assistive device. TDMA slots in a Channel 1 single FDM channel User 1 User 2 Each slot has a length of 250 ms User 3 Channel 2 User 1 Rotate slot by 50ms till conflict User 2 resolved Channel 125 User 1 e Th re area a total of 125 FDM channels User 2 Date rate of up to 1 Mbps Figure 6: The customized protocol for multiple users in crowd mode. The choice of the transmission slot follows a uniform dis- the user to implement frequency hopping spread spectrum tribution as shown in Equation (11), and in case, a device fails (FHSS) [23]. Since a pair of Arduinos is used for data pro- to communicate for 2 seconds in crowd mode, then the ran- cessing and communication, frequency hopping is difficult domly generated time slots are shifted to 50 ms repeatedly till to implement. This is due to the slight difference in the Ardu- successful communication at both ends is achieved. Upon ino pair clocks as well as the susceptibility of this low-cost successful communication and built in acknowledgment platform to the temperature changes. This practically results from the receiver, the message is repeated continuously in in a clock drift and the transceivers getting out of the allocated time slot which consists of a start of data iden- synchronization. tifier (unique for each pair) and data. Figure 6 depicts a gen- The effectiveness of the assistive device in crowd mode is eral scenario for crowd mode operation of assistive device. shown in MATLAB simulation. In Figure 7, the simulation The chip also supports frequency hopping thus enabling for number of collisions vs. the number of devices is given. S 8 Applied Bionics and Biomechanics 200 400 0 50 100 150 200 250 300 350 Samples Ankle sensor 0 50 100 150 200 250 300 Number of devices With customized TDMA With FDMA only S.D with FDMA only Mean: customized TDMA Mean: with FDMA only 0 50 100 150 200 250 300 350 Figure 7: Number of devices vs. number of collisions. Samples Up-45° Normal Down-45° Figure 9: Actual sensor measurements after passing through a moving average filter (inclined obstacle up ahead). 4. Experiments and Results 0 100 200 300 400 500 600 700 800 In order to verify the working of device, it is tested while Samples moving on a staircase and under an inclined roof, i.e., under Ankle sensor a staircase. Both the belt and ankle module are used, and the scenario is developed such that the user moves on plain sur- face towards the stair, then he moves on the stairs and at the 300 end he reaches platform. The subject’s height for each sce- nario was 175 cm with the waist sensors installed at a height of 100 cm from the ground and ankle sensors installed at 20 cm from the ground. The average stride length was mea- sured and averaged 32 cm. In each experiment, the outputs of all sensors are passed 0 100 200 300 400 500 600 700 800 through a moving average filter before using them for a deci- Samples sion. The outputs from these sensors after passing through Up-45° the moving average filter are depicted in Figures 8 and 9. In Normal the first scenario depicted in Figure 8, the subject is climbing Down-45° stairs. The sensor outputs can be seen to properly register the shape of stairs. The 45 -up sensor is also registering the stairs; Figure 8: Actual sensor measurements after passing through a moving average filter (stairs). however, it is the roof of the staircase which also happened to be inverted stairs. In case of any obstacle classification, the algorithm is designed to gather a data of at least 20 samples and then decide and inform the subject about the upcoming It can be seen when the device operates in crowd mode, the obstacle type when the distance approaches 300 cm or less. number of collisions significantly decreases. The results show Figure 9 shows the user walking with an incline obstacle how adapting proposed crowd mode technique improved the upfront. It can be seen that the shape of obstacle has been results several times as compared to the normal mode. properly registered by all the ultrasonic sensors and the dis- Another possible solution to this problem is using a GPS tance given by each sensor, other than 45 -down sensor, clock for synchronization of the hopping time pattern. This decrease as the user moves. solution can also benefit in the integration of maps with the In order to test performance and effectiveness of the device and providing the user with a clearer picture of the device in real life, statistical experiments are performed. For surroundings. However, due to cost overhead, this idea will experimental purpose, a setup is arranged in a 610 × 365 be considered for future modifications. cm room with static obstacle randomly scattered and three Distance (cm) Distance (cm) Number of collisions Distance (cm) Distance (cm) Applied Bionics and Biomechanics 9 1) 2) 3) 4) 1) 2) 3) 4) Type of assistive device Type of assistive-device (a) (b) Figure 10: Statistical results: (a) number of cotillions and (b) distance covered. (1) Without any aid, (2) with cane, (3) with belt module only, and (4) with belt and ankle module. taken, during each set of experiment. It can be seen that using 0.8 both belt and ankle devices resulted in covering of more dis- Collision point 0.6 tance while distance covered without any aid is the least. 0.4 In order to analyze the working of device for moving object, we considered case of a vehicle moving in reverse 0.2 direction and the user of the device standing behind it. The collected data is analyzed in MATLAB. The optimal value –0.2 for sampling time of T =0:2 s was selected using a hit and 01 2 3 4 56 7 8 9 864 2 0 –0.4 trial method. The speed used for the subject is 4.5 km/h Time (s) Time (s) –0.6 [24], whereas the velocity of the object is average reverse –0.8 speed of a car calculated in campus parking. It was observed that for ultrasonic sensor, with 400 cm range, the response –1 0 500 1000 1500 2000 2500 3000 3500 4000 time was about 1 s, which was too small for the state estima- x- axis (cm) tor to converge for final range time. The same experiment was repeated by replacing ultra- Vehicle trajectory Vehicles current position sonic sensor with a LADAR Lite v3 sensor. The maximum Subject with assitive device distance used between the moving object and the subject Subjects current position was chosen corresponding to the maximum range of the LADAR Lite v3 sensor which is 40 m with a three-sigma Figure 11: Collision trajectory of the moving object and subject bound of 3 cm in each direction. The trajectories of the vehi- with proposed device. cle in reverse and a subject with the proposed device are depicted in Figure 11. In Figure 12, we have given the root people walking very slowly in it to simulate dynamic obsta- mean square error (RMSE) of the range, range rate, and the cles. A person with covered eyes is moved in the room for five time to react using a Kalman filter. Since the time to react minutes. Different sets of experiments, repeated 10 times, are becomes shorter, the warning beep interval is reduced to warn the subject about an imminent collision with the mov- performed; firstly, without any assistive aid; secondly, with a white cane; thirdly, with assistance from only the belt mod- ing object. However, it is better than that of ultrasonic sensor. ule, and at the end with assistance from both the belt and the ankle modules. The distance, in terms of steps, covered 5. Conclusion Remarks by the person during each experiment along with the number of collisions is calculated separately. In this paper, we have given the design and construction of a In Figure 10(a), we have given the statistical results of new assistive device for visually impaired people. It is of low number of collisions in different experiments, plotted in the cost and uses ultrasonic sensors and IMU to obtain informa- form of boxplot. It can be seen from the plot that the number tion about surroundings. Unlike other such devices that uses of collisions is least when combination of belt and ankle ultrasonic sensors, this device has more features where it can modules is used. On the other hand, by using only the belt distinguish position of obstacles as well as can detect different module, the number of collisions is slightly more than com- types of obstacles at different positions in front of the user. bined belt-ankle modules. This means that if only the belt The device can detect obstacle at head level, in front, and at module is used, cost can be reduced with expense of slightly ground. At the end, experiments are done to check the statis- more collisions. On the other hand, in Figure 10(b)), we have tical advantages of the new device. It was seen that the newly given the accumulative distance covered, in term of steps constructed assistive device performs better as compared to y- axis (cm) Number of colisions Number of steps taken 10 Applied Bionics and Biomechanics 0.015 0.8 0.010 0.6 0.005 0.4 0.2 0 0 123 45 67 8 9 0 123 45 67 8 9 Time (s) Time (s) Estimated time to react Range (r) Range rate (RR) (a) (b) Figure 12: Root mean square errors from the estimation process for trajectories depicted in Figure 11. (a) Range and range rate. (b) Time to react. Table 1: Comparison of range-based assistive device with the device presented in the paper. Author [14] [15] [16] [17] [25] [26] Our device Range Small Medium Medium Medium Medium Large Medium Cost High Low High High Low High Low Weight Heavy Light Heavy Heavy Light Heavy Light Obstacle detection Front Front Front Front Front Front Front-up-down Stair detection Yes No No No No No Yes Instructions No No No No No No Yes conventional assistive methods. The modular form of the References device has increased the choice for the user. The user can [1] WHO Team: Blindnesws, World report on vision, Editor: only purchase one module at lesser price and lose only few World Health Organization, License: CC BY-NC-SA 3.0 features. The other module can also be added in the same first IGO, Geneva, Switzerland, 2019. module after some time. The use of voice instruction was a [2] H. Sekiguchi and H. Nakayama, “On a history and a present new method of communication with the user, in which clear circumstances of walking aid for persons with visual impair- instructions can be added by the user in his/her own voice. ment in Japan,” 5th International Conference on Civil Engi- In order to get better response time for fast moving neering, 2002. objects, the front ultrasonic sensor was replaced by LADAR [3] S. Sue, “Assisting the blind and visually impaired: guidelines Lite v3 sensor. It was observed that the LADAR gave better for eye health workers and other helpers,” Community eye results. However, this has increased the cost of the device. health/International Centre for Eye Health, vol. 16, no. 45, The comparison of existing range sensor-based assistive pp. 7–9, 2003. devices for visually impaired people with our device is given [4] M. Gori, G. Cappagli, A. Tonelli, G. Baud-Bovy, and in Table 1. It can be observed that our device provides better S. Finocchietti, “Devices for visually impaired people: high features than other devices. In the near future, camera for technological devices with low user acceptance and no adapt- vision-based assistance will be added in the proposed device. ability for children,” Neuroscience & Biobehavioral Reviews, vol. 69, pp. 79–88, 2016. Data Availability [5] W. Elmannai and E. Khaled, “Sensor-based assistive devices for visually-impaired people: current status, challenges, and The data can be provided on demand. future directions,” Sensors, vol. 17, no. 3, p. 565, 2018. [6] S. S. Senjam, “Assistive technology for students with visual dis- Conflicts of Interest ability: classification matters,” Kerala Journal of Ophthalmol- ogy, vol. 31, no. 2, p. 86, 2019. The authors declare no conflict of interest regarding this [7] B. Mocanu, R. Tapu, and T. Zaharia, “When ultrasonic publication. sensors and computer vision join forces for efficient obstacle detection and recognition,” Sensors, vol. 16, no. 11, p. 1807, Acknowledgments [8] N. Kanwal, E. Bostanci, K. Currie, and A. F. Clark, “A naviga- This work was supported in by the Technology Development tion system for the visually impaired: a fusion of vision and Fund (Grant No. TDF02-203), Higher Education Commis- depth sensor,” Applied bionics and biomechanics, vol. 2015, sion, Pakistan. 16 pages, 2015. R & RR (cm & cm/s) (s) Applied Bionics and Biomechanics 11 [25] A. L. Petsiuk and J. M. Pearce, “Low-cost open source [9] D. Nakamura, H. Takizawa, M. Aoyagi, N. Ezaki, and S. Mizuno, “Smartphone-based escalator recognition for the ultrasound-sensing based navigational support for the visually visually impaired,” Sensors, vol. 17, no. 5, p. 1057, 2017. impaired,” Sensors, vol. 19, no. 17, p. 3783, 2019. [10] K. Eunjeong and K. Eun, “A vision-based wayfinding system [26] C. Ton, A. Omar, V. Szedenko et al., “LIDAR assist spatial for visually impaired people using situation awareness and sensing for the visually impaired and performance analysis,” IEEE Transactions on Neural Systems and Rehabilitation Engi- activity-based instructions,” Sensors, vol. 17, no. 8, p. 1882, 2017. neering, vol. 26, no. 9, pp. 1727–1734, 2018. [11] A. Brilhault, S. Kammoun, O. Gutierrez, P. Truillet, and C. Jouffrais, “Fusion of artificial vision and GPS to improve blind pedestrian positioning,” in Proceedings of the 4th IFIP International Conference on New Tech-nologies, Mobility and Security (NTMS), pp. 1–5, Paris, France, 2011. [12] B. R. Prudhvi and R. Bagani, “Silicon eyes: GPS-GSM based navigation assistant for visually impaired using capacitive touch braille keypad and smart SMS facility,” in Proceedings of the 2013 World Congress on Computer and Information Technology (WCCIT), pp. 22–24, Sousse, Tunisia, 2013. [13] M. F. Saaid, I. Ismail, and M. Z. H. Noor, “Radio frequency identification walking stick (RFIWS): A device for the blind,” in Proceedings of the 5th International Colloquium on Signal Processing & Its Applications, Kuala Lumpur, Malaysia, 2009. [14] N. A. Ayat, A. F. Mahmoud, and F. S. Ahmed, “Assistive infra- red sensor based smart stick for blind people,” in IEEE Science and Information Conference (SAI), London, UK, 2015. [15] V. Surapol and K. Nambunmee, “iSonar: an obstacle warning device for the totally blind,” Journal of AssistiveRehabilitative & Therapeutic Technologies, vol. 2, no. 1, p. 23114, 2014. [16] S. Aymaz and C. Tugrul, “Ultrasonic assistive headset for visu- ally impaired people,” in 39th IEEE International Conference on Telecommunications and Signal Processing, Vienna, Aus- tria, 2016. [17] T. Pallejà, M. Tresanchez, M. Teixidó, and J. Palacin, “Bioin- spired electronic white cane implementation based on a LIDAR, a tri-axial accelerometer and a tactile belt,” Sensors, vol. 10, no. 12, pp. 11322–11339, 2010. [18] K. Chaccour, J. Eid, R. Darazi, A. H. el Hassani, and E. Andres, “Multisensor guided walker for visually impaired elderly peo- ple,” in IEEE International Conference on Advances in Biomed- ical Engineering, Beirut, Lebanon, 2015. [19] H. Ahmed, I. Ullah, U. Khan et al., “Adaptive filtering on GPS- aided MEMS-IMU for optimal estimation of ground vehicle trajectory,” Sensors, vol. 19, no. 24, p. 5357, 2019. [20] I. Ullah, T. L. Song, and T. Kirubarajan, “Active vehicle protec- tion using angle and time-to-go information from high- resolution infrared sensors,” Optical Engineering, vol. 54, no. 5, article 053110, 2015. [21] Systems, e, nRF24L01 / 2.4GHz RF / products / home - ultra low power wireless solutions from Nordic semiconductor, 2018, June 2018, http://www.nordicsemi.com/eng/Products/2.4GHz-RF/ nRF24L01. [22] C. Zhurong, H. Chao, L. Jingsheng, and L. Shoubin, “Protocol architecture for wireless body area network based on nRF24L01,” in IEEE International Conference on Automation and Logistics, pp. 3050–3054, Qingdao, China, 2008. [23] A. B. Forouzan, Data Communications & Networking, Tata McGraw-Hill Education, New York, NY, USA, 2006. [24] D. D. Clark, A. D. Heyes, and C. I. Howarth, “The efficiency and walking speed of visually impaired people,” Ergonomics, vol. 29, no. 6, pp. 779–789, 1986. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Bionics and Biomechanics Hindawi Publishing Corporation

Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person

Loading next page...
 
/lp/hindawi-publishing-corporation/construction-and-analysis-of-a-novel-wearable-assistive-device-for-a-Fb29H1Qzc5
Publisher
Hindawi Publishing Corporation
Copyright
Copyright © 2020 Shahid Akram et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
ISSN
1176-2322
eISSN
1754-2103
DOI
10.1155/2020/6153128
Publisher site
See Article on Publisher Site

Abstract

Hindawi Applied Bionics and Biomechanics Volume 2020, Article ID 6153128, 11 pages https://doi.org/10.1155/2020/6153128 Research Article Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person 1 1 2 1 1 Shahid Akram, Ali Mahmood, Ihsan Ullah, Muhammad Tahir Mujtabah, Ali Bin Yasin, 1 3 1 Asif Raza Butt, Muhammad Shafique, and Sajjad Manzoor Department of Electrical Engineering, Mirpur University of Science and Technology (MUST), Mirpur, 10250 AJK, Pakistan Department of Electrical Engineering CUI, Abbottabad Campus, Abbottabad, KPK, Pakistan Department of Biomedical Engineering, Riphah International University, Islamabad, Pakistan Correspondence should be addressed to Sajjad Manzoor; sajjad.ee@must.edu.pk Received 26 December 2019; Revised 9 July 2020; Accepted 19 September 2020; Published 15 October 2020 Academic Editor: Mohammad Rahimi-Gorji Copyright © 2020 Shahid Akram et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. In this paper, we have given the design and development of a new wearable device that assists visually impaired individuals to travel independently and confidently. The newly proposed device is based on range-based sensors and would work effectively in both indoor and outdoor conditions. It is constructed in the form of two separate modules; one module is designed such that it can be attached to the waist belt of the user, and the other module is designed to wear it on ankle of the user. Both the modules communicate with each other using wireless communication and can cover the full front environment of the user. The information about the front environment is transmitted to the user, via headphone, by sending a set of voice instructions, stored in a memory card added in the belt module. In order to use the device in crowd mode, appropriate networking techniques were also implemented in the prototype such that the interference of two or more devices in the close vicinity can be avoided. In the end, effectiveness of the device is analyzed and proved by conducting experiments and obtaining statistical results. [4–6]. In Figure 1, different assistive devices for the blind 1. Introduction are shown, which can be helpful during the motion of the With the advances in technology, commercial hardware and user. The ETAs use sensors to detect obstacles in front of software applications are developed to make life easy for peo- their users and give information about the front environment ple with physical weakness. At least 2.2 billion people of the and guide the user in a manner that they would safely move forward. In some other devices, cameras are used for world have a vision impairment or blindness [1]. They may belong to a category, with moderately visually impairment, vision-based assistance of their user during motion. In some severely visually impairment, or totally blind. Most of the cases, Raspberry Pi cameras [7] are used to detect objects in visually impaired people move by using conventional front, and in other cases, Kinect sensors [8] are used to calcu- methods, i.e., white canes, guide dogs, tactile paving, and in late the distance of the user from the obstacles. Smartphone some other cases move with the help of another person, camera-based elevator finder application was developed in called as sight guide [2, 3]. With the evolution of technology, [9]. This would help users to find if they are moving in the there is an immense need for the development of easy-to-use direction of the elevator or not. Mobile phone camera, with devices that would be helpful for the visually blind people in color codes was used in [10] in order to convey information mobility. In this way, visually impaired individuals may about the inside of a building. In [11], the authors have used travel independently with confidence and participate in daily fusion of artificial vision and GPS for locomotion assistance activities. and obstacle detection for the user. Fusion of ultrasonic sen- Recently, many types of electronics travel aids (ETA) sor, GPS, and GSM was introduced in a silicon glove in [12]. have been developed for mobility assistance of blind people A walking stick that uses radiofrequency identification (RFI) 2 Applied Bionics and Biomechanics (a) (b) (c) (d) Figure 1: Different mobility assistive devices for visually impaired people: (a) [18], (b) [11], (c) [15], and (d) [7]. to solve all the issues, mentioned above, in this paper, we was constructed in [13] in order to help blind people navigate on their sidewalk. However, most of the assistive device for have proposed an assistive device for the blind which is inex- blind uses range-based sensors, which are cost effective, have pensive and lighter in weight. It is a wearable modular device. good availability, and easy to operate. These user friendly It is equipped with range-based sensors. These modules can devices use IR sensors [14], for small range; ultrasonic sen- be attached to the waist belt and the ankle of the user. This device covers more area in front of the user and is effectively sors [15, 16], for medium range; and/or LIDAR [17], for long range, to detect and localize obstacles in front of the user. helpful for his mobility. The experimental analysis on the Most of the assistive devices for the blind, developed, till device is also done to check its effectiveness. now, are either very costly or bulky for the user to wear or This paper is further organized as follows: in Section 2, hold. The conventional white cane is already heavy and addi- the purpose for the design and development, along with con- tion of any device on it can make it more difficult to hold. siderations taken during design phase of the new device for Vision-based devices, with cameras attached to them, mostly visually impaired person, are given. The architecture as well belong to the categories that are both heavy and expensive. as the components used in the device is also discussed in this These days, small cameras are available; however, cost section. The algorithm designed for working of the device is remains the matter of concern in them. Similarly, smart- given in Section 3. The model for the state estimator is dis- phones, which are used in some applications for assistance cussed in Section 3.1. In Section 3.2, we have considered for visually impaired, are also expensive. The devices based the issue of communication within the device in the crowd on ultrasonic sensor or any other range-based sensors do mode, i.e., crowded environment with multiple users at same not cover all the features of the front environment. In order place. In Section 4, the experimental and statistical results of Applied Bionics and Biomechanics 3 The belt module (BM) is designed in such a way that it the device are given, so that the effectiveness of the device for visually impaired people can be evaluated. Finally, in Section can be attached to the waist belt of the user. The architecture 5, the conclusion of the paper is given. The expected future of the belt module is shown in Figure 3. It serves as the master development in the proposed device is also discussed in this module for the ankle module (AM), and it has its own section. rechargeable power supply. This module has three ultrasonic sensors targeted in different directions. The first sensor is directed normal to the user body, and it would detect any 2. An Assistive Device for Visually Impaired obstacles in front of the user. The second sensor is directed People (aVIP) in upward direction at an angle of 45 from first sensor. This sensor can detect obstacles only at head level such as lower Design, construction, and development of a new assistive roof or lower inclined side of the stairs. A third sensor is device for visually impaired people are given in this section. adjusted in such a way that it is directed downward at an One of the intentions of the research is to develop a cost- angle of -45 from the first sensor. It can detect any pits or effective assistive device for visually impaired people. hole in the ground in front of the user. It would also detect any obstacle of small size or step, not detected by the first sen- 2.1. Design Consideration for Assistive Device. During the sor. However, due to its orientation, the sensor would only be design phase of the device, the following considerations were able to detect small obstacles such as step or footpath unless made to get maximum information about the front the user reaches very close to it. In combination with the environment: ankle module this sensor would also detect small, hollow obstacles, i.e., tables and chairs. The developed belt module (1) The device should determine all obstacles in front of is given in Figure 4(a). the user body, from the ground to the head, as shown The belt module has two, commercially available, Ardu- in Figure 2(a). It should be able to distinguish obsta- ino Mini microprocessors, connected with each other. This cles at the following locations: is due to the reason that each Arduino Mini has only one SPI interface while the belt module needs two SPI interfaces, (i) Large obstacles in front of the user, as shown in i.e., one for a SD card that stores audio instructions and other Figure 2(b) for a communication device. The memory card is connected (ii) Holes and pits, on the ground, in front of the to one of the microprocessors. This memory card would have user, as shown in Figure 2(c) audio instructions recorded in it, each as a separate audio file. A headphone is also connected to the device through an (iii) Hanging or inclined obstacles at the level of the audio amplifier. An nRf24L01 module acting as a receiver is head of the user, as shown in Figure 2(d) connected to the other microprocessor in the belt module, (iv) Small object or stairs in front of the user, as so that the information from the ankle module can be wire- shown in Figure 2(e) lessly received. The architecture of the ankle module is also shown in (v) Hollow objects (i.e., tables and chairs) in front of Figure 3. It has a single Arduino Mini microprocessor con- the user nected to a rechargeable battery. It has one ultrasonic sensor in it which is directed in the direction perpendicular to the (2) Information about the environment, in front, should user body. This sensor can detect all the obstacles of small be given in the form of a clear voice instructions. size, which are undetectable to the sensors in the belt module. (3) The instructions should be easy to understand; this An inertial measurement unit (IMU) sensor is attached to the could be recorded in the user’s own voice. module to detect the dynamics of leg motion and differenti- ate straight and bent leg during swing or stance phase of In Figure 2(a), the layout of the proposed device is given motion. Using the IMU sensor, only obstacles in front of that would accomplish all of the abovementioned tasks. A the user would be detected and the ground would not be con- modular device, consisting of two modules with four differ- sidered as an obstacle when the leg would be in bent position. ent proximity sensors, is proposed. More detail of the archi- It can also be used to count the number of steps taken by user. tecture of the proposed device is discussed in Section 2.2. The data from the ankle module is transmitted to the belt module through the nRF24L01 module. The nRF24L01 mod- 2.2. Architecture of aVIP Device. In order to cover the maxi- ule in the belt module acts as a receiver while the one in the mum area in front of the user, the device is designed in a ankle module acts as a transmitter. The constructed ankle modular form by combining sensors, commercially available module is given in Figure 4(b)). microprocessors, transmitter, and receiver. The architecture of the device is given in Figure 3. It is subdivided into two 3. Working of the Device and Algorithm for modules that communicate with each other without any wire. Obstacle Detection These modules are named as There are three ultrasonic sensors installed at waist height to (i) Belt module measure the distance to the ground or an obstacle. One is (ii) Ankle module installed normal to the body of user, the other up waist at 4 Applied Bionics and Biomechanics For head Waist level belt obstacles module For large obstacles in front For pits in ground in front Ankle module For small obstacles in front (a) (b) (c) (d) (e) Figure 2: Evaluation for design consideration of an assistive device for visually impaired people. (a) Layout of the device. (b) Obstacle at head level. (c) A pit hole on a straight path. (d) Obstacle upfront. (e) Stairs or small obstacle in front. ° ° All the sensors perform differently under different ter- 45 , and the third down waist at an angle of negative 45 -down to normal. The device installed on the ankle has an ultrasonic rains and scenarios. Some of the common scenarios that a sensor directed normal to the body as well as an IMU. The belt subject may face have been evaluated and considered in the and ankle modules detect the obstacle separately. However, design of the hardware, as shown in Figures 2(b)–2(e). The only the belt module has an SD card connected to it that con- outputs from all the sensors are used to warn the user about tains audio instructions for the user about the nature of the any obstacle ahead. The first sensor to register this kind of obstacles in front. Furthermore, headphones are also con- obstacle will be the normal sensor. In case of a hole on a nected to belt module. Therefore, the belt module acts as the leveled surface, the primary action or warning is given to master and the ankle module acts as a slave. the user based on the input from the waist down sensor; Applied Bionics and Biomechanics 5 Initial Anckle module setting Ultrasound Ultrasound SD – memory sensor sensor card Arduino Mini Audio amplifier Arduino Mini Arduino IMU sensor Battery Mini Battery Headphone NRF24L01 NRF24L01 transmitter reciever Belt module Figure 3: Architecture for modular assistive device for visually impaired people. (a) (b) Figure 4: Newly constructed modular assistive device: (a) belt module and (b) ankle module. however, the ankle sensor with IMU sensor output can be a user wears a device for the first time, it should be operated used to navigate the obstacle and scan the obstacle for clear- in setting mode to adjust these limits. During the setting ance. In the scenario of the leveled surface with obstacle at mode, the user is instructed to stand on a plane surface and head level only, up waist sensors gives distance. While in hold a plane paper sheet at head level. By pressing setting the case of stairs or small obstacle ahead, the normal sensor switch, the device would automatically adjust obstacle detec- is again the first sensor to register an obstacle, but the classi- tion distance limits l ðlimitÞ and l ðlimitÞ for the head and h g fication of the obstacle can only be performed by the down ground levels. In order to avoid false detection due to uneven waist sensor and the ankle sensor. ground, small clearances ϵ , ϵ , and ϵ are added in the algo- 1 2 3 The outputs of sensors are passed through a moving aver- rithm. Once the setting is done, the setting mode is closed age filter before using them for a decision. Since the moving and the device is operated on working mode. For instanta- average filter is also a low-pass filter, it smoothens any abrupt neous values l , l , and l of the respected normal, head level, b h g transitions thus eliminating any noise or erroneous measure- ground level sensors in the belt module and values l of ankle ments. The moving average filter implemented in the hard- sensor, the algorithm for operation of the assistive device is ware is depicted in Figure 5. The assistive device has two given in Algorithm 1. modes of operation: setting mode and working mode. Same obstacle detection distance limits l ðlimitÞ and l ðlimitÞ, for b a 3.1. The State Estimation Process. Since the measurements, the front sensor in the belt and ankle modules, respectively, especially for moving obstacles, are of stochastic nature, a can be adjusted for the users of different heights. Therefore, Kalman Filter (KF) can be deployed for target state estima- there is no need to change these parameters. However, obsta- tion. The measurement from the sensor consists of range R, cle detection distance limits l ðlimitÞ and l ðlimitÞ for the h g to the moving object, i.e., z = R the time to react, τ, is not respective head and ground sensors in the belt module can estimated for a few initial scans; afterwards, it is initialized be affected by the height of each user. Therefore, whenever from the estimated range and range rate and is made part 6 Applied Bionics and Biomechanics the estimation part can be written as follows, where first Input + Output we compute the innovation covariance matrix by D χ χ 1 ξ = H Γ H + R, ð7Þ k k k/ðÞ k−1 k where R is the measurement variance and is computed from the three-sigma bound of the sensor. The Kalman gain is given by χ −1 Δ = Γ H + ξ , ð8Þ k k/ k−1 D ðÞ k k such that, the estimated state vector can now be calcu- lated from the Kalman gain and the measurement residue as Figure 5: The moving average filter used as a low-pass filter for the raw sensor input. χb = χb + Δ z − H χb : ð9Þ k k k/k k/ðÞ k−1 k k/ðÞ k−1 of the filtering process. The final state propagates in discrete Finally, the state covariance matrix is computed by the domain by following equation: χ = ϕχ + U + w , ð1Þ k k−1 k k Γ = I − Δ H Γ : ð10Þ k/k k k/ k−1 k ðÞ where χ is the state vector, U = ½00 T and the plant k k The plant covariance matrix is initialized based on the noise, w , is assumed to be white Gaussian noise with zero process explained in [20]. The quality of the estimate mean. The plant noise is characterized by a known covari- depends on the sampling time T along with the assumed ance matrix Q. The state vector consisting of the range, R, measurement process noise covariance matrix R and process range rate R, and time to react, τ,isdefined as noise variance q. _ ð2Þ χ = : R R τ 3.2. Crowd Mode Communication. The nRF24L01 chip is used for communication purpose, between two modules of The state transition matrix ϕ in Equation (1) is given as the constructed assistive device [21]. It is a radio transceiver with a frequency operating range of 2.4-2.5 GHz in the ISM 2 3 1 T 0 band. This chip was used due to its low cost, size, and low 6 7 power consumption as compared to other available options 6 7 ϕ = 01 0 , ð3Þ 4 5 [22]. Each transceiver can use 125 channels with a channel switching time less than 200 ms and data rate of up to 00 1 1 Mbps. This implies that 125 different devices can operate in the same environment without interfering with each other. with T being the sampling time. The plant noise covari- The assistive device for the blind, given in paper, was first ance matrix Q can be expressed as developed for assistance of little children at schools for blind. 2 3 4 2 Therefore, it can have more than 125 users, which can cause T T problems in communication. Thus, a remedy for limitation 6 7 4 2 6 7 in the number of communication channels is required. The 6 7 6 T 7 2 number of devices operating in an environment however Q = q: , ð4Þ 6 7 T 0 6 7 can be increased by utilizing time division multiple access 6 7 4 5 8 (TDMA) [23], which however, is not supported by the 00 nRF2401 chip. The device is made to operate in two modes, the first one being the normal mode and the second one crowd mode. The crowd mode utilizes a custom nonstandard where process noise variance is denoted by q. The KF update TDMA technique. Each device can transmit for 250 ms only and prediction equations as given in literature [19] can be whereas the receiver listens throughout and receives data expressed as from the transmitter already registered with it. The Probabil- ity density function of the uniformly distributed slot interval χb = ϕχb + U + w : ð5Þ k k k/k−1 ðÞ k−1 /ðÞ k−1 on an interval with a width (b-a) of 250 ms is given in terms of the Heaviside step function by With HsðÞ − a −HsðÞ − b b b Ps ðÞ = : ð11Þ Γ = ϕΓ ϕ + Q, ð6Þ k/k−1 ðÞ k−1 /ðÞ k−1 b − a 3 Applied Bionics and Biomechanics 7 (1) Manual setting of distance limit l ðlimitÞ in belt module for obstacle normal to the user, and distance limit l ðlimitÞ in ankle module for obstacle in front direction of the user. (2) Setting mode: If setting button is on; (a) l ðlimitÞ = current distance calculated by 45 -up sensor. (b) l ðlimitÞ= current distance calculated by 45 -down sensor. (3) Operating mode: if setting button is off; (I) If l ðlimitÞ Instruction 1: “Obstacle in front” (II) If l < l ðlimitÞ & l > l ðlimitÞ a a b b Instruction 2: “Small obstacle in front” (III) If l < l ðlimitÞ h h Instruction 3: “Obstacle at head-level” (IV) If l > l ðlimitÞ + ϵ g g 1 Instruction 4: “Pit in ground” (V) If l < l ðlimitÞ + ϵ & l < l ðlimitÞ g g 2 a a Instruction 5: “Small obstacle or footpath ahead” (VI) If l < l ðlimitÞ − ϵ & l > l ðlimitÞ g g 3 a a Instruction 6: “Table or chair ahead” Algorithm 1. Algorithm for working of the proposed assistive device. TDMA slots in a Channel 1 single FDM channel User 1 User 2 Each slot has a length of 250 ms User 3 Channel 2 User 1 Rotate slot by 50ms till conflict User 2 resolved Channel 125 User 1 e Th re area a total of 125 FDM channels User 2 Date rate of up to 1 Mbps Figure 6: The customized protocol for multiple users in crowd mode. The choice of the transmission slot follows a uniform dis- the user to implement frequency hopping spread spectrum tribution as shown in Equation (11), and in case, a device fails (FHSS) [23]. Since a pair of Arduinos is used for data pro- to communicate for 2 seconds in crowd mode, then the ran- cessing and communication, frequency hopping is difficult domly generated time slots are shifted to 50 ms repeatedly till to implement. This is due to the slight difference in the Ardu- successful communication at both ends is achieved. Upon ino pair clocks as well as the susceptibility of this low-cost successful communication and built in acknowledgment platform to the temperature changes. This practically results from the receiver, the message is repeated continuously in in a clock drift and the transceivers getting out of the allocated time slot which consists of a start of data iden- synchronization. tifier (unique for each pair) and data. Figure 6 depicts a gen- The effectiveness of the assistive device in crowd mode is eral scenario for crowd mode operation of assistive device. shown in MATLAB simulation. In Figure 7, the simulation The chip also supports frequency hopping thus enabling for number of collisions vs. the number of devices is given. S 8 Applied Bionics and Biomechanics 200 400 0 50 100 150 200 250 300 350 Samples Ankle sensor 0 50 100 150 200 250 300 Number of devices With customized TDMA With FDMA only S.D with FDMA only Mean: customized TDMA Mean: with FDMA only 0 50 100 150 200 250 300 350 Figure 7: Number of devices vs. number of collisions. Samples Up-45° Normal Down-45° Figure 9: Actual sensor measurements after passing through a moving average filter (inclined obstacle up ahead). 4. Experiments and Results 0 100 200 300 400 500 600 700 800 In order to verify the working of device, it is tested while Samples moving on a staircase and under an inclined roof, i.e., under Ankle sensor a staircase. Both the belt and ankle module are used, and the scenario is developed such that the user moves on plain sur- face towards the stair, then he moves on the stairs and at the 300 end he reaches platform. The subject’s height for each sce- nario was 175 cm with the waist sensors installed at a height of 100 cm from the ground and ankle sensors installed at 20 cm from the ground. The average stride length was mea- sured and averaged 32 cm. In each experiment, the outputs of all sensors are passed 0 100 200 300 400 500 600 700 800 through a moving average filter before using them for a deci- Samples sion. The outputs from these sensors after passing through Up-45° the moving average filter are depicted in Figures 8 and 9. In Normal the first scenario depicted in Figure 8, the subject is climbing Down-45° stairs. The sensor outputs can be seen to properly register the shape of stairs. The 45 -up sensor is also registering the stairs; Figure 8: Actual sensor measurements after passing through a moving average filter (stairs). however, it is the roof of the staircase which also happened to be inverted stairs. In case of any obstacle classification, the algorithm is designed to gather a data of at least 20 samples and then decide and inform the subject about the upcoming It can be seen when the device operates in crowd mode, the obstacle type when the distance approaches 300 cm or less. number of collisions significantly decreases. The results show Figure 9 shows the user walking with an incline obstacle how adapting proposed crowd mode technique improved the upfront. It can be seen that the shape of obstacle has been results several times as compared to the normal mode. properly registered by all the ultrasonic sensors and the dis- Another possible solution to this problem is using a GPS tance given by each sensor, other than 45 -down sensor, clock for synchronization of the hopping time pattern. This decrease as the user moves. solution can also benefit in the integration of maps with the In order to test performance and effectiveness of the device and providing the user with a clearer picture of the device in real life, statistical experiments are performed. For surroundings. However, due to cost overhead, this idea will experimental purpose, a setup is arranged in a 610 × 365 be considered for future modifications. cm room with static obstacle randomly scattered and three Distance (cm) Distance (cm) Number of collisions Distance (cm) Distance (cm) Applied Bionics and Biomechanics 9 1) 2) 3) 4) 1) 2) 3) 4) Type of assistive device Type of assistive-device (a) (b) Figure 10: Statistical results: (a) number of cotillions and (b) distance covered. (1) Without any aid, (2) with cane, (3) with belt module only, and (4) with belt and ankle module. taken, during each set of experiment. It can be seen that using 0.8 both belt and ankle devices resulted in covering of more dis- Collision point 0.6 tance while distance covered without any aid is the least. 0.4 In order to analyze the working of device for moving object, we considered case of a vehicle moving in reverse 0.2 direction and the user of the device standing behind it. The collected data is analyzed in MATLAB. The optimal value –0.2 for sampling time of T =0:2 s was selected using a hit and 01 2 3 4 56 7 8 9 864 2 0 –0.4 trial method. The speed used for the subject is 4.5 km/h Time (s) Time (s) –0.6 [24], whereas the velocity of the object is average reverse –0.8 speed of a car calculated in campus parking. It was observed that for ultrasonic sensor, with 400 cm range, the response –1 0 500 1000 1500 2000 2500 3000 3500 4000 time was about 1 s, which was too small for the state estima- x- axis (cm) tor to converge for final range time. The same experiment was repeated by replacing ultra- Vehicle trajectory Vehicles current position sonic sensor with a LADAR Lite v3 sensor. The maximum Subject with assitive device distance used between the moving object and the subject Subjects current position was chosen corresponding to the maximum range of the LADAR Lite v3 sensor which is 40 m with a three-sigma Figure 11: Collision trajectory of the moving object and subject bound of 3 cm in each direction. The trajectories of the vehi- with proposed device. cle in reverse and a subject with the proposed device are depicted in Figure 11. In Figure 12, we have given the root people walking very slowly in it to simulate dynamic obsta- mean square error (RMSE) of the range, range rate, and the cles. A person with covered eyes is moved in the room for five time to react using a Kalman filter. Since the time to react minutes. Different sets of experiments, repeated 10 times, are becomes shorter, the warning beep interval is reduced to warn the subject about an imminent collision with the mov- performed; firstly, without any assistive aid; secondly, with a white cane; thirdly, with assistance from only the belt mod- ing object. However, it is better than that of ultrasonic sensor. ule, and at the end with assistance from both the belt and the ankle modules. The distance, in terms of steps, covered 5. Conclusion Remarks by the person during each experiment along with the number of collisions is calculated separately. In this paper, we have given the design and construction of a In Figure 10(a), we have given the statistical results of new assistive device for visually impaired people. It is of low number of collisions in different experiments, plotted in the cost and uses ultrasonic sensors and IMU to obtain informa- form of boxplot. It can be seen from the plot that the number tion about surroundings. Unlike other such devices that uses of collisions is least when combination of belt and ankle ultrasonic sensors, this device has more features where it can modules is used. On the other hand, by using only the belt distinguish position of obstacles as well as can detect different module, the number of collisions is slightly more than com- types of obstacles at different positions in front of the user. bined belt-ankle modules. This means that if only the belt The device can detect obstacle at head level, in front, and at module is used, cost can be reduced with expense of slightly ground. At the end, experiments are done to check the statis- more collisions. On the other hand, in Figure 10(b)), we have tical advantages of the new device. It was seen that the newly given the accumulative distance covered, in term of steps constructed assistive device performs better as compared to y- axis (cm) Number of colisions Number of steps taken 10 Applied Bionics and Biomechanics 0.015 0.8 0.010 0.6 0.005 0.4 0.2 0 0 123 45 67 8 9 0 123 45 67 8 9 Time (s) Time (s) Estimated time to react Range (r) Range rate (RR) (a) (b) Figure 12: Root mean square errors from the estimation process for trajectories depicted in Figure 11. (a) Range and range rate. (b) Time to react. Table 1: Comparison of range-based assistive device with the device presented in the paper. Author [14] [15] [16] [17] [25] [26] Our device Range Small Medium Medium Medium Medium Large Medium Cost High Low High High Low High Low Weight Heavy Light Heavy Heavy Light Heavy Light Obstacle detection Front Front Front Front Front Front Front-up-down Stair detection Yes No No No No No Yes Instructions No No No No No No Yes conventional assistive methods. The modular form of the References device has increased the choice for the user. The user can [1] WHO Team: Blindnesws, World report on vision, Editor: only purchase one module at lesser price and lose only few World Health Organization, License: CC BY-NC-SA 3.0 features. The other module can also be added in the same first IGO, Geneva, Switzerland, 2019. module after some time. The use of voice instruction was a [2] H. Sekiguchi and H. Nakayama, “On a history and a present new method of communication with the user, in which clear circumstances of walking aid for persons with visual impair- instructions can be added by the user in his/her own voice. ment in Japan,” 5th International Conference on Civil Engi- In order to get better response time for fast moving neering, 2002. objects, the front ultrasonic sensor was replaced by LADAR [3] S. Sue, “Assisting the blind and visually impaired: guidelines Lite v3 sensor. It was observed that the LADAR gave better for eye health workers and other helpers,” Community eye results. However, this has increased the cost of the device. health/International Centre for Eye Health, vol. 16, no. 45, The comparison of existing range sensor-based assistive pp. 7–9, 2003. devices for visually impaired people with our device is given [4] M. Gori, G. Cappagli, A. Tonelli, G. Baud-Bovy, and in Table 1. It can be observed that our device provides better S. Finocchietti, “Devices for visually impaired people: high features than other devices. In the near future, camera for technological devices with low user acceptance and no adapt- vision-based assistance will be added in the proposed device. ability for children,” Neuroscience & Biobehavioral Reviews, vol. 69, pp. 79–88, 2016. Data Availability [5] W. Elmannai and E. Khaled, “Sensor-based assistive devices for visually-impaired people: current status, challenges, and The data can be provided on demand. future directions,” Sensors, vol. 17, no. 3, p. 565, 2018. [6] S. S. Senjam, “Assistive technology for students with visual dis- Conflicts of Interest ability: classification matters,” Kerala Journal of Ophthalmol- ogy, vol. 31, no. 2, p. 86, 2019. The authors declare no conflict of interest regarding this [7] B. Mocanu, R. Tapu, and T. Zaharia, “When ultrasonic publication. sensors and computer vision join forces for efficient obstacle detection and recognition,” Sensors, vol. 16, no. 11, p. 1807, Acknowledgments [8] N. Kanwal, E. Bostanci, K. Currie, and A. F. Clark, “A naviga- This work was supported in by the Technology Development tion system for the visually impaired: a fusion of vision and Fund (Grant No. TDF02-203), Higher Education Commis- depth sensor,” Applied bionics and biomechanics, vol. 2015, sion, Pakistan. 16 pages, 2015. R & RR (cm & cm/s) (s) Applied Bionics and Biomechanics 11 [25] A. L. Petsiuk and J. M. Pearce, “Low-cost open source [9] D. Nakamura, H. Takizawa, M. Aoyagi, N. Ezaki, and S. Mizuno, “Smartphone-based escalator recognition for the ultrasound-sensing based navigational support for the visually visually impaired,” Sensors, vol. 17, no. 5, p. 1057, 2017. impaired,” Sensors, vol. 19, no. 17, p. 3783, 2019. [10] K. Eunjeong and K. Eun, “A vision-based wayfinding system [26] C. Ton, A. Omar, V. Szedenko et al., “LIDAR assist spatial for visually impaired people using situation awareness and sensing for the visually impaired and performance analysis,” IEEE Transactions on Neural Systems and Rehabilitation Engi- activity-based instructions,” Sensors, vol. 17, no. 8, p. 1882, 2017. neering, vol. 26, no. 9, pp. 1727–1734, 2018. [11] A. Brilhault, S. Kammoun, O. Gutierrez, P. Truillet, and C. Jouffrais, “Fusion of artificial vision and GPS to improve blind pedestrian positioning,” in Proceedings of the 4th IFIP International Conference on New Tech-nologies, Mobility and Security (NTMS), pp. 1–5, Paris, France, 2011. [12] B. R. Prudhvi and R. Bagani, “Silicon eyes: GPS-GSM based navigation assistant for visually impaired using capacitive touch braille keypad and smart SMS facility,” in Proceedings of the 2013 World Congress on Computer and Information Technology (WCCIT), pp. 22–24, Sousse, Tunisia, 2013. [13] M. F. Saaid, I. Ismail, and M. Z. H. Noor, “Radio frequency identification walking stick (RFIWS): A device for the blind,” in Proceedings of the 5th International Colloquium on Signal Processing & Its Applications, Kuala Lumpur, Malaysia, 2009. [14] N. A. Ayat, A. F. Mahmoud, and F. S. Ahmed, “Assistive infra- red sensor based smart stick for blind people,” in IEEE Science and Information Conference (SAI), London, UK, 2015. [15] V. Surapol and K. Nambunmee, “iSonar: an obstacle warning device for the totally blind,” Journal of AssistiveRehabilitative & Therapeutic Technologies, vol. 2, no. 1, p. 23114, 2014. [16] S. Aymaz and C. Tugrul, “Ultrasonic assistive headset for visu- ally impaired people,” in 39th IEEE International Conference on Telecommunications and Signal Processing, Vienna, Aus- tria, 2016. [17] T. Pallejà, M. Tresanchez, M. Teixidó, and J. Palacin, “Bioin- spired electronic white cane implementation based on a LIDAR, a tri-axial accelerometer and a tactile belt,” Sensors, vol. 10, no. 12, pp. 11322–11339, 2010. [18] K. Chaccour, J. Eid, R. Darazi, A. H. el Hassani, and E. Andres, “Multisensor guided walker for visually impaired elderly peo- ple,” in IEEE International Conference on Advances in Biomed- ical Engineering, Beirut, Lebanon, 2015. [19] H. Ahmed, I. Ullah, U. Khan et al., “Adaptive filtering on GPS- aided MEMS-IMU for optimal estimation of ground vehicle trajectory,” Sensors, vol. 19, no. 24, p. 5357, 2019. [20] I. Ullah, T. L. Song, and T. Kirubarajan, “Active vehicle protec- tion using angle and time-to-go information from high- resolution infrared sensors,” Optical Engineering, vol. 54, no. 5, article 053110, 2015. [21] Systems, e, nRF24L01 / 2.4GHz RF / products / home - ultra low power wireless solutions from Nordic semiconductor, 2018, June 2018, http://www.nordicsemi.com/eng/Products/2.4GHz-RF/ nRF24L01. [22] C. Zhurong, H. Chao, L. Jingsheng, and L. Shoubin, “Protocol architecture for wireless body area network based on nRF24L01,” in IEEE International Conference on Automation and Logistics, pp. 3050–3054, Qingdao, China, 2008. [23] A. B. Forouzan, Data Communications & Networking, Tata McGraw-Hill Education, New York, NY, USA, 2006. [24] D. D. Clark, A. D. Heyes, and C. I. Howarth, “The efficiency and walking speed of visually impaired people,” Ergonomics, vol. 29, no. 6, pp. 779–789, 1986.

Journal

Applied Bionics and BiomechanicsHindawi Publishing Corporation

Published: Oct 15, 2020

References