A Deep Neural Network Model for Upper Limb Swing Pattern to Control an Active Bi

Thisara PATHIRANA, Hiroshan GUNAWARDANE, Nimali T MEDAGEDARA

(1. Department of Mechanical Engineering, Faculty of Engineering Technology,The Open University of Sri Lanka, Nugegoda, Sri Lanka;2. Department of Mechanical Engineering, Faculty of Applied Science,The University of British Columbia, Vancouver, Canada)

Abstract: Leg amputations are common in accidents and diseases. The present active bionic legs use Electromyography (EMG) signals in lower limbs (just before the location of the amputation) to generate active control signals. The active control with EMGs greatly limits the potential of using these bionic legs because most accidents and diseases cause severe damages to tissues/muscles which originates EMG signals. As an alternative, the present research attempted to use an upper limb swing pattern to control an active bionic leg. A deep neural network (DNN) model is implemented to recognize the patterns in upper limb swing, and it is used to translate these signals into active control input of a bionic leg. The proposed approach can generate a full gait cycle within 1082 milliseconds, and it is comparable to the normal (a person without any disability) 1070 milliseconds gait cycle.

Keywords: Active Bionic Leg, Deep Neural Network, Human Gait Cycle Time

1 Introduction

An amputation is a surgical removal of all or a part of a limb due to various diseases and accidents.Limb amputations can be categorized into two main categories based on its severity. Minor amputations are the amputation of different minor parts in a limb (e.g.,fingers) and major amputations are both below-knee and above-knee amputations which severely damage and disable the lower limb (e.g., remove entire lower limb). Major amputations have been classified in ISO 8549-2:1989[1]. In general, mechanical prosthetic legs without any sophisticated controlling parts have been used for amputations below the knee. However, major knee amputations required active prosthetic legs with advanced controlling units to generate the required controlled movements. Alternatively, passive prosthetic legs are used to support the static stability for the disabled persons.

Most of the passive prosthetic designs have been developed for below knee (BK, transtibial) amputations[2], however, these designs are unable to restore the body’s dynamic balance during the bipedal walking[2-5]. Therefore, active bionic legs are used to actively control the dynamics of the bionic legs[6-8].Advanced controlling methods have been developed in the contemporary research to restore the proper dynamic behavior to restore the locomotion[9-10]. The commercially available bionic legs are ineffectual where the leg amputation has occurred along with inactivation of the muscles of the thigh, which mainly challenges in responding to aggregated EMG signals emitted from the thigh.

This research discusses the design and implementation of an active bionic leg, based on the pre-trained model using DNN to predict the knee motion from the arm swinging pattern. DNN is used as a random function approximation tool to estimate the most effective and ideal solutions, while defining computing functions or distributions[11]. The movement of the arm is maintained by the muscle groups in the upper limb and the movements can be detected by measuring EMG signals. It is a common practice to measure motor signals to measure specific movements.But it is challenging to use EEGs to decode the upper limb arm swinging patterns[11]. Therefore, in the present research, an Inertial Measurement Unit (IMU)and EMG sensors were used to measure the arm swinging.

The paper is organized as follows, Section 2 discusses the literature and the background of the present work, Section 3 elaborates the proposed methodology, Section 5 presents the results, and Section 6 states the conclusion of this research and its future directions.

2 Literature Review

2.1 Active Bionic Legs

Most of the active prosthesis are operated by EMG, EEG, and gesture control signals. The signals generated in the intact muscles in the lower limb are mainly used in EMG based active bionics legs[9]. Five muscles in lower limb (Gluteus Maximus, Gluteus Medias, Vasti, Soleus, and Gastrocnemius) are mainly contributing to the human locomotion. These muscles control the accelerations of the center of mass within the vertical, fore aft, and mediolateral directions whilst human beings stroll and run at their desired speeds[12-13]. Vastus intermedius, Vastus Lateralis,Vastus medialis and Rectus femoris contribute majorly for the knee motion[14]. Electrodes placed on these muscles can measure EMG signals for controlling active bionic leg[14,15].

Previously designed passive prosthetic legs are focused in stabilizing the knee movements. Transformable prosthetic legs were developed according to Inoue et al., 2016 for walking in regular surfaces. This design is mainly focused on establishing stance phase.They strategically applied the external load to the knee and minimized the unnecessary knee flexion moment,thereby they attempted to avoid unnecessary knee flexion during activities such as stair climbing[16].However, this design cannot adapt the changes in the surface of the stair. Most state-of-the-art designs do not exhibit an angle and torque relationship and also unable to adjust their mechanics for stairs and ramps.Shepherd and Rouse, 2017 developed a quasi-passive ankle-foot prosthesis with a customizable torque-angle curve and an ability to rapidly manipulate ankle stiffness between tasks[17]. The customizable torque-angle path is obtained through a transmission based on cams and fiberglass leaf springs. In order to achieve variable stiffness, a small motor is used to actively adjust the leaf spring[17]. In this design they have considered the starting point of the swing phase with the rapid movement of the knee joint. But the design is not automatically adjusting to individual differences. Murabayashi and Inoue, 2020 have developed a passive leg mechanism that helps to run[18]. Most of the active bionic legs were designed to control using motor signals. Furuya et al., 2013 developed an active prosthetic leg using a linear actuator with a helical mover and a stator[19]. The movements of the knee joint were achieved using a high helical motor which use magnetic levitation[19]. One of the main disadvantages of this system is high power consumption.

New active bionic legs use both EMG and EEG signals[20]. These legs are mainly focused on providing voluntary movements with motorized joints, improving gait cycle, improving the adaptability to complex terrains, and eliminating undesirable movements that may unbalance the body[14].

2.2 Input Signals

In general, EEG-based systems are used when most of the thigh muscles are inactive. The use of EMGs in these situations are limited because poor electrode conductivity, increased signal conditioning complexity, and poor signal strength (less than 100μV)[21]. However, EMGs are more reliable than EEG and can be detected around 100μV– 90mV within the DC frequency range of 0–10kHz[21,22]. These signals are carefully measured and preprocessed before feature extraction.

2.3 Deep Learning in Active Bionic Legs

DNN-based programs are developed to predict the angle of shoulder flexion/extension and abduction/adduction movements using bio signals[23,24-28].Pre-developed sEMG signals can be used as predictors in these systems. IEMG (integrated EMG) signals and SSC (number of slope sign changes) were used as features to predict the angles[26,29]. A similar, approach is used in this paper to generate active control signals for the bionic leg.

3 Methodology

The development of the bionic leg is carried out in two stages. Initially, DNN algorithm was developed with the data from healthy human. Secondly leg motion was controlled by using classical control techniques according to the input signal which comes from DNN.This DNN based bionic leg is intelligent to predict the knee motion related to the arm motion of human, during bipedal walking. The main challenge of this research is to eliminate the knee motion for unwanted arm movement, which was achieved by considering the rate of change of the arm angle during the bipedal walking for operating DNN. The multilayer DNN was trained under the backpropagation algorithm. Human arm swinging data were tracked by sensors and corresponding lower limb data such as thigh angle and knee angle were tracked in the same time during the bipedal walking, in natural speed by using wearable sensors (Fig.1). Developed leg was validated using the mechanical and control engineering theories and practices[30-33]. The model was developed for walking on flat surfaces.

3.1 Data Acquisition

Data acquisition system was able to collect training data using EMG Sensors and accelerometer for training the DNN. Fig. 1 illustrate the attachment of the wearable sensor which function in combination of the IMU sensors and EMG sensors. Data acquisition system was programmed by using microcontroller, and sensorized data were processed using a computer with data processing algorithm. EMG and IMU sensors were attached to the microcontroller for expediting the data acquisition during the bipedal walking. Microcontroller and microcontroller’s boot loading platform are based on the Arduino Mega which powered by ATmega2560 AVR (AVR is a family of microcontrollers developed by Atmel beginning in 1996). Microcontroller used has 16 MIPS CPU speed at 16 MHz and operates between 4.5-5.5 volts. Data acquisition part was executed on selected microcontroller with combination of the EMG sensor and eight IMU sensors that comes from one wire data bus. This help to smoothen the data for neural network development.The Exponential filter is a recursive filter, and this filter is used for calculating a new, smoothed value by using the last smoothed value and a new measurement[34]. Two successive data samples are used to filter the data in the exponential filter algorithm during the real time data.

The small execution delay for the filtering process between raw data and filtered data as Fig. 2 were existed by using the exponential filter algorithm. Delay time of the filtering process was eliminated by using exponential filter and filtering smoothness can be adjusted for each application using the weight factor with having Exponential term.

Fig.1 Wearable Sensor Attachment on Human Body for the Data Acquisition

Fig.2 Raw Data and Filtered Data - Arm Swinging

3.2 Function Estimation between Motor Angle and Knee Joint Angle

According to the design of Bionic leg, Knee joint was operated by the motor. Motor connected with lever mechanisms operates the knee joint to reduce the applied torque of the motor in addition to transmit motion.Predicted knee joint angle which comes from DNN according to the hand motion, should maintain the motor angle. Motor angle for corresponding knee angle was calculated by mathematical model which was governed from 3D simulation of the bionic leg as shown in Fig. 3, 4, 5, 6 and 7.

3D prototype was used to demonstrate the kinematics system shown in Fig.3 and the mathematical function was developed between knee angle and motor angle using curve fitting techniques. 3D prototype for the knee mechanism in Fig.3, was developed using “SolidWorks” Software and was imported to “MATLAB” software using “Simscape Multibody ToolBox” in MATLAB as Fig. 4.

Mechanical kinematics system in Fig.3 was controlled by providing the input for revolutejoint using sinusoidal input in Simulink block diagram as in Fig. 4.Input angle vs time to the motor and corresponding output angle vs time form knee joint were represented on Fig. 5 and 6 respectively.

By considering the input and output in Fig.5 and 6,the experimental curve on Fig. 7 was fitted using MATLAB and equation was derived using curve fitting for the combination of θ1and θ2as,

Estimated curve can be smoothened by increasing the degree of equation.θ2can be estimated using knee joint angle(θ1) which comes from DNN for the knee joint control task using equation 1. θ2was used to input command to the DC motor PID Controller.

Fig.3 “Simscape” Mechanics Explorer Output Window CAD Leg Model

Fig.4 Simulink Block Diagram for Kinematics Control

Fig.5 Input Signal to Motor (Angle vs Time Graph)

4 Deep Neural Net work model

4.1 DNN Architecture

Four inputs and one output were applied on the deep neural network to manipulate the bionic leg’s knee. Inputs of the DNN in Fig.8 werethe body weight,body height,rate of movement of arm and IMU angle value on the arm during walking.For a particular person,body weight and body height have constant values and rate of movement of arm and IMU angle values are changed with time during the walking with arm swinging.Trained DNN is capable for predicting the knee angle of the bionic leg when the person walking with arm swinging.

Fig.6 Output Angle Form Knee Joint(Anglevs Time Graph)

Fig.7 Experimental and Estimated Angles Output

Fig.8 DNN Architecture with 4 Inputs and 1 Output

Fig.8 shows the DNN architecture for operating the correct movement of the leg after transmitting the predicted kneeangleto PID controller of the bionic leg.This architecture was created using the“Google TensorFlow™” Machine learning tool library and“Python” Programming language. Supervised learning was used to train the Neural Network. Multilayer flat(2D) DNN was trained using backpropagation method.The neurons in each dense layers were adjusted by considering the prediction accuracy and losses during the several number of training programs.

4.2 Architecture Optimization

DNN was developed by using random architecture with random hyper-parameters (weight, bias, layers, etc.).This architecture was optimized for obtainingbetter results.

Developed DNN has 98% accuracy for the architecture in Fig. 8.

4.3 DNN Validation

The K-Fold Cross-Validation method was used in this research. The data set was split into 70% training and 30% testing using the train-test-split command.Function was fitted by the function approximation tool for the training set and the function estimation tool was used to predict the model outputs for the data in the testing set (testing sets was never applied for training).The generated error losses were compiled for calculating mean absolute testing error losses and used to evaluate the model.

5 Results

For equation (2), 1070ms for average gait cycle time of healthy human, 4ms for execution and communication delays and 7.95ms was estimated for predicted knee joint controlling delay using PID control algorithm and hardware devices which were used.From equation (2), final gait cycle time for bionic leg(BL) was estimated as,

Complete prototype of bionic leg was represented in Fig.9(b). The wearable component which represented in Fig.9(a) was used to track the arm motion to provide DNN for the knee angle prediction.

Terminal Stance and Pre-Swing phase in human gait cycle were represented with the movement of bionic leg during the gait cycle (Fig.10).Arm data curve with corresponding actual knee motion during the gait cycle and predicted data form the Deep Neural Network (DNN) (DNN.knee) is shown in Fig.10.

According to the graph in Fig. 10, percentage error between actual knee (ActualKnee) angle and knee angle in Bionic Leg (BioLeg.Knee) for 100 working cycles was found to be 15.76%.

Deviation of the variance data are described as standard deviations of the actual knee, predicted knee and Bionic leg knee as 1.755, 1.872, 1.820 and 1.915 degrees respectively.

Fig.9 DNN Based Bionic Leg Model

Fig.10 Terminal Stance and Pre-swing Phase in Gait Cycle for Bionic Leg, Collected Data and Predicted Data

6 Conclusion

This bionic leg model was developed for upper knee amputation with inactive muscles. Currently commercially available bionic legs unable to detect any data from inactive muscle using EMG. EEG based bionic leg developmentsare available to accomplish meet this challenge, but this solution is not convenient due to the difficulty of the signal detection and signal conditioning. Human bipedal walking is rhythmic activity that control form the CPG of the body. This research developed DNN model to rehabilitate lower-limb gait pattern using bionic leg. Deep Neural Network model has capability to predict the lower-limb gait pattern from the arm swinging pattern. This research introduces the data analysis to train a DNN model by using effective sensor modules and expressed proper data types as the features of the model. At the end of this research, prototype bionic leg was developed using Deep Neural network (DNN) model and knee joint control algorithm by controlling knee joint according to the arm swinging pattern. Developed bionic leg has the capability to predict the corresponding knee joint angle using human arm swinging pattern with the execution delay of 12 milliseconds. By considering the gait cycle, the deviation of the time delay with a healthy person is 1.11%. The existing system is acting using supervised learning approach including DNN. For future developments, this prototype bionic leg can be developed using reinforcement learning system. This system will be trained an identical walking pattern for bionic leg. Therefore, the arm swinging movement would not be needed every time during the gait cycle with bionic leg after training for the reinforcement learning system.

Acknowledgment

My heart felt gratitudeis expressed to Soft Robotics Research Group of the Faculty of Engineering Technology of, The Open University of Sri Lanka.Specifically, I would like to extend my gratitude for Samith Wijesinghe, Mininidu Sheheran, Nadun Kaluarachchi and Vijayanadan Rishan are mentioned for their valuable assistance in this research project.