ArtificialIntelligence Rehabilitation Evaluation and Training System for Degener

LIU Weichen,SHEN Mengqi,ZHANG Anda,CHENG Yiting,ZHANG Wenqiang,

(1.Academy for Engineering and Technology,Fudan University,Shanghai 200433,China；2.School of Computer Science,Fudan University,Shanghai200433,China)

Abstract:Degeneration of joint disease is one of the problems that threaten global public health.Currently,the therapies of the disease are mainly conservative but not very effective.To solve the problem,we need to find effective,convenient and inexpensive therapies.With the rapid development of artificial intelligence,we innovatively propose to combine Traditional Chinese Medicine(TCM)with artificial intelligence to design a rehabilitation assessment system based on TCM Daoyin.Our system consists of four subsystems:the spine movement assessment system,the posture recognition and correction system,the background music recommendation system,and the physiological signal monitoring system.We incorporate several technologies such as keypoint detection,posture estimation,heart rate detection,and deriving respiration from electrocardiogram(ECG)signals.Finally,we integrate the four subsystems into a portable wireless device so that the rehabilitation equipment is well suited for home and community environment.The system can effectively alleviate the problem of an inadequate number of physicians and nurses.At the same time,it can promote our TCM culture as well.

Keywords:rehabilitation;Traditional Chinese Medicine;artificial intelligence;degeneration of joint disease

1 Introduction

T oday,many policies support the inheritance and development of Traditional Chinese Medicine(TCM).It is popular that manufacturing a portable artificial intelligence(AI)rehabilitation evaluation and training system to improve the rehabilitation ability and promote the rehabilitation equipment industry of TCM.Degeneration of joint disease(DJD)[1]is a physiological and pathological degeneration process that occurs in the spine as the human body naturally ages.DJD can cause a variety of spinal-related disease syndromes and bring pain and stiffness,which will seriously affect patients’daily life.Severely,patients’nervous systems may be compressed and cause paralysis.Today,DJD has become one of the serious public health problems.The main clinical rehabilitation methods are traction therapy,infrared hyperthermia,percutaneous electrical stimulation,etc.,but they are only suitable for some Grade A tertiary hospitals and specialist rehabilitation hospitals because of expensive and large equipment.In general,the doctor will advise the patient to take some conservative treatment unless the invasive surgical treatment must be taken.Therefore,we need to find an appropriate therapy as well as manufacture an efficient and inexpensive device for the therapy of DJD.

At present,there are various clinical treatments for DJD.HU[2]believes that moxibustion can relieve pain,replenish Qi and thus treat DJD.YANG et al.[3]believe that“Jin Gu Bing Ju,Chan Xuan Xiang Ji”and advocate the use of LI’s Tuina method to treat DJD.The authors in Ref.[4–5]demonstrate that the combination of TCM and electromagnetic wave irradiation for the treatment of DJD has more prominent efficacy.The authors in Ref.[6–7]demonstrate that the combination of laser irradiation and physical traction therapy also performs better on the treatment of DJD,but it is only suitable for some Grade A tertiary hospitals and specialist rehabilitation hospitals because of expensive and large equipment.

Among all kinds of conservative therapies,TCM has a 5 000-year history of development for health care.TCM Daoyin[8]is guided by TCM theories such as Yin and Yang,five elements,meridians,and internal organs.It promotes functional recovery through breathing and exhalation,physical activities,and psychological regulation.It also has obvious therapeutic effects on the rehabilitation and prevention of soft tissue and bone and joint diseases.It is increasingly used in clinical treatment.Therefore,we choose to establish an artificial intelligence rehabilitation system based on TCM Daoyin for the therapy of DJD.

At the same time,we construct a background music database based on the five-tone theory of TCM[9].It provides suitable background music for patients as an aid to therapy when practicing TCM Daoyin.TCM’s five tones are“Gong Shang Jue Zhi Yu”,the ancient way to recognize the sound,which originally belongs to the category of temperament.Huangdi Neijing(the Medical Classic of the Yellow Emperor)introduces five tones into TCM theory and forms a certain system.Fivetone therapy is based on the theory that the five tones correspond to the five organs in Huangdi Neijing.Studies[9–10]have shown that the five-tone therapy has a significant effect.Therefore,we embed the background music database into our system for the therapy of DJD.

In this paper,we propose a portable wireless system device that innovatively combines artificial intelligence and TCM for the rehabilitation of DJD.It replaces part of repetitive heavy physical labor in traditional rehabilitation.Furthermore,it realizes automatic,accurate,and intelligent rehabilitation and is very suitable for family and community clinics because of its small size,wireless,and portability.What’s more,it could reduce the stress of physicians and rehabilitation trainers and increase the flexibility and effectiveness of patient rehabilitation training.Therefore,our system has strong innovation and practicability.

The remaining structure of this paper is as follows.Section 2 presents an overall overview of our system and describes the functions implemented in each subsystem separately.Section 3 elaborates the implementation principles,calculation formulas,and algorithm details for each of our subsystems.Section 4 describes the overall system integration application and the overall system workflow.

2 Functions

In this paper,we innovatively propose to develop a smallscale rehabilitation assessment and training system by combining the popular technologies in the field of artificial intelligence with TCM Daoyin.The system can provide comprehensive automatic assessment and assisted exercises for patients with DJD during their rehabilitation training.We will integrate all the modules of the system into a Mini-PC and transmitted the data via 5G signals to facilitate real-time rehabilitation training for patients and remote monitoring and guidance for doctors.As shown in Fig.1,the system includes the following four subsystems:

1)SMA:spine movement assessment system.It automatically evaluates the range of motion of human cervical and lumbar vertebrae joints.

2)PRC:posture recognition and correction system.It can automatically compare and correct errors between patient actions and standard expert actions.

3)BMR:background music recommendation system.It filters and builds a library of TCM five-tone background music.

4)PSM:physiological signal monitoring system.It monitors and alerts the patient’s respiratory rate and heart rate stability.

2.1 SMA:Spine Movement Assessment System

The spine movement assessment system is designed by combining popular techniques in computer vision.It can automatically measure the maximum joint mobility of the patient’s cervical and lumbar vertebrae before the start and after the end of the patient’s TCM Daoyin rehabilitation training.The joint mobility includes 6 types of subscales:cervical extension,lumbar extension,cervical lateral bending,lumbar lateral bending,cervical rotation,and lumbar rotation.Therefore,this subsystem can visually and quantitatively assess the effect of the patient’s rehabilitation training.

First,the system takes an image of the patient in a specific computational scenario and uses maximum force to stretch the cervical or lumbar vertebrae.Then,the system performs image processing operations and inputs the results into a neural network model for posture estimation and inference.As a result,we can obtain information about the 2D skeletal coordinates corresponding to the patient image.We use different calculation or estimation formulas for three different types of joint mobility.Finally,we combine the three metric perspectives to give the patient’s rehabilitation evaluation score.

▲Figure 1.Intelligent rehabilitation assessment and training system

2.2 PRC:Posture Recognition and Correction System

The function of the posture recognition and correction system is to identify and classify the movements of patients in real time,while they are practicing the TCM Daoyin.The system will compare and score each Daoyin movement with the standard movements of the expert group and provide real-time feedback.Finally,the patient’s overall movements are analyzed when the patient has completed the entire Daoyin set.The system will give a score for each segment of different movements and an overall score for the entire set.

The system uses a neural network model to identify and classify the current rehabilitation Daoyin movements being practiced by the patient based on a feature learning approach.It will get the patient’s historical posture sequence by tracking the key points of the patient’s body at the same time.What’s more,we collect and construct a small sample dataset to train the network models.Then,the system automatically segments the patient’s historical pose sequence based on the patient’s action category by the method of sequence similarity estimation.Finally,we obtain the similarity score of the patients’Daoyin segments by comparing each segment of the patient’s posture sequence with the standard posture sequence of the expert.

2.3 BMR:Background Music Recommendation System

Many clinical studies show that TCM’s five-tone therapy can effectively assist or even directly contribute to the rehabilitation process of patients.The overall rehabilitation efficiency of patients treated with the TCM five-tone therapy has been significantly improved.

Most scholars think that“Gong Shang Jue Zhi Yu”corresponds to“Do,Re,Mi,So,La”in the modern numbered musical notation from a musical point of view.However,traditional music has indistinct tuning and frequent modulation.There are many different versions of the same piece and the tuning is inconsistent with each other.The current research application of five-tone therapy is too limited and lacks a holistic approach.Moreover,there is no systematic and standardized five-tone music dataset.The manual-based five-tone classification is laborious and inefficient because there is a huge amount of online music.Therefore,our background music recommendation system aims at addressing the shortcomings of the current clinical application of five-tone therapy.It uses computer audition technology to automatically filter and classify music from a large library of music online.Eventually,a five-tone database of background music for patients to perform TCM Daoyin exercises is constructed to assist patients’rehabilitation training.

2.4 PSM:Physiological Signal Monitoring System

Doctors often monitor patients’health based on some of their physiological signatures such as the heart rate and respiratory rate.We need to monitor the heart rate and respiratory rate in real time while the patient is practicing TCM Daoyin training,ensure the patient’s safety when practicing alone,and alerts the patient’s family and doctor in case of abnormal conditions.The physiological signal monitoring system is based on a biosensor that uses a single-lead approach to acquire the patient’s ECG signal.Then,the collected ECG signal data is further processed to achieve accurate and reliable heart rate detection.

For the respiratory rate,there is no non-invasive respiratory rate monitoring device available in the market.Traditional methods use large instruments with low accuracy,which are difficult to extend to home rehabilitation equipment.The respiratory rate is also an important physiological signal for the patient,so we have to find a method for detecting respiratory rate that would work with our mobile rehabilitation system.The physiological signal monitoring system captures the patient’s respiratory rate using an algorithm,named ECG derived respiration(EDR),which extracts the derived respiratory signal from the ECG signal in real time.The system thus enables real-time monitoring of the heart rate and respiratory rate of patients during TCM Daoyin training.

3 Algorithms

3.1 Algorithms of SMA System

3.1.1 Human Posture Estimation

Human posture estimation is to estimate the human pose by calculating the relative position of the human key points in 3D space and correctly linking the human key points that have been detected in the picture.The key points of the human body usually correspond to the joints with degrees of freedom of the human body,for example,neck,shoulder,elbow,wrist,knee,ankle,etc.,as shown in Fig.2.

CAO et al.[11]proposed the OpenPose multi-person pose estimation framework.It is a real-time approach to detect the 2D pose of multiple people in an image.We use this model to implement the estimation and scoring of the patient’s spinal motion status.The model can automatically identify and match the key points of the patient’s body in each frame of the received image.It consists of four main components:feature extraction,partial affinity fields(PAFs)prediction,key point location confidence map prediction,and pose graph matching.First,we input an image and extract features by convolutional neural networks(CNNs)to get a set of feature maps.Then we use CNNs to extract part confidence maps and part affinity fields respectively.Second,we use bipartite matching in graph theory to find the part association,which connects the joints of the same person.Finally,we merge them into the overall skeleton of a person.Details of the algorithm can be found in Ref.[11].

3.1.2 Calculation of Joint Mobility

Because the overall system is designed for simplicity and lightness,the hardware equipment conditions are somewhat limited.Under the condition of only one camera,we use different calculation or estimation formulas for different joint mobility.

▲Figure 2.Common key points of the human body

As shown in Fig.3,we use precise positioning to calculate the angles for the extension and lateral bending by the coordinates of the body’s key points.Here we set the coordinate origin at the Midhip point.As shown in Table 1,we have run experiments and ensured that the error is within 3°,which allows us to score the patient’s posture very well.

1)Cervical and lumbar extension angles

We use different formulas to calculate the extension angles of the patient’s cervical and lumbar spine.

As for the patient’s cervical extension angle,we calculate it from the coordinates of the nose and ear key points in the lateral view.

As for the patient’s lumbar extension angle,we calculate it from the coordinates of the neck and midhip key points in the lateral view.

▲Figure 3.(a)Extension angles;(b)lateral bending angles;(c)rotation angles

▼Table 1.Results of our spine movement assessment(SMA)system

where,∝leis the lumbar extension angle,x iis the horizontal coordinate of key pointi,andy iis the vertical coordinate of key pointi.

2)Cervical and lumbar lateral bending angles

We capture the patient’s key points from the front and back of the patient.Accordingly,we calculate the patient’s cervical and lumbar lateral bending angles.

As for the patient’s cervical lateral bending angle,we calculate it from the coordinates of the nose key points and estimate eye_center key points in the front view.

The estimated coordinates of the eye_center key point are shown as follows.

where,x iis the horizontal coordinate of key pointi,andy iis the vertical coordinate of key pointi.

Calculation of the patient’s cervical lateral bending angle is shown as follows.

where∝clis the cervical lateral bending angle,x iis the horizontal coordinate of key pointi,andy iis the vertical coordinate of key pointi.

As for the patient’s lumbar lateral bending angle,we calculate it from the estimated coordinates of the 7th cervical spinous process and the 5th lumbar spinous process key points in the back view.

We use the neck key point and the estimated mid_ear key point,where the mid_ear key point’s calculation method is the same as the eye_center key point,to estimate coordinates of the 7th cervical spinous process key point.

wherex iis the horizontal coordinate of key pointiandy iis the vertical coordinate of key pointi.Hereu=0.5 is used to calculate the 7th cervical spinous process.

We use the neck key point and the midhip key point to estimate coordinates of the 5th lumbar spinous process key point.

wherex iis the horizontal coordinate of key pointiandy iis the vertical coordinate of key pointi.Herev=0.2 is used to calculate the 5th lumbar spinous process.

Calculation of the patient’s lumbar lateral bending angle is shown as follows.

where,∝llis the lumbar lateral bending angle,x iis the horizontal coordinate of key pointi,andy iis the vertical coordinate of key pointi.

3)Cervical and lumbar rotation angles

For the rotation angle，due to the limitation of the number of cameras condition,we take the estimation approach.

We assume that the patient’s cervical spine is rotated by 90°and the angle between the nose key point in line with the midpoint of the Lshoulder and Rshoulder key points，and the Lshoulder key point in line with the Rshoulder key point denoted byε,ε90=45°.Disn_msis the distance between the horizontal coordinate of the nose key point and the horizontal coordinate of the Lshoulder and Rshoulder key points’midpoint.And Disls_rsis the distance between the horizontal coordinate of the Lshoulder and Rshoulder key points.It can be calculated that Disn_msis a quarter of Disls_rs.

where,x iis the horizontal coordinate of key pointi.

Estimation of the rotation angle of the cervical spine is shown as follows.

where∝cris the cervical rotation angle.

As for the lumbar rotation angle,we simply replace the Lshoulder and Rshoulder key points with the Lhip and Rhip key points.We give no further explaination to keep the paper reasonably concise.

3.2 Algorithms of PRC System

The system first applies the same OpenPose multi-person pose estimation framework[11]as in Section 3.1 to estimate the patient’s posture while training the TCM Daoyin.After that,we use the pose sequence tracking and classification algorithm and the pose sequence segmentation and similarity calculation algorithm to evaluate the patient’s practice posture.Meanwhile,to improve the robustness of the overall network,we add a view adaptive(VA)module[12]to the bottom layer of the network.The VA module performs a two-dimensional transformation of the input pose utilizing the learned rotation matrix parameters and translation matrix parameters.

3.2.1 Pose Sequence Tracking and Classification Algorithm

Almost all existing video action classification algorithms take the whole video as input and get the classification result by a trained deep network.However,since the input must include the whole sequence,even if the inference speed of the deep network can reach real time,this does not meet the demand of real-time classification.In the real-time sign language detection task,Google proposes a lightweight sign language detection network based on pose recognition[13].It can classify each frame of the video signal in real time.We borrow this idea to classify and track the patient training video in real time by using patient pose information and inter-frame pose optical flow information for each frame[14].The human pose optical flow information obtained by computing inter-frame based on pose estimation is input to the long short-term memory(LSTM)[15].It obtains realtime classification results at each frame based on the current inter-frame optical flow characteristics and historical optical flow characteristics.The structure of our network is shown in Fig.4.

3.2.2 Pose Sequence Segmentation and Similarity Calculation

We obtain the category of each frame of the patient’s training video by the algorithm of Section 3.2.1.Then,we segment the video subsequences belonging to the same category to obtain the pose sequence of the patient’s current exercise.By the method of similarity calculation we can get the similarity between the patient’s current pose sequence and the standard pose sequence.Finally,we score and correct the patient’s movements in real time.

Because of the difficulty in aligning the patient’s current pose sequence with the standard pose sequence,we use a dynamic time warping(DTW)algorithm[16].It finds the best alignment path with the lowest total sequence cost by dynamic programming.

where Cost(i,j)is the alignment cost of pointiand pointj,andD(i,j)is the distance between pointiand pointj.

After obtaining the alignment path,we calculate the score of the segment based on the key points identified during the patient’s practice sequence and the corresponding key points of the standard sequence.Since the movements of each video vary greatly,we choose different key points of attention and different calculation methods for different movements.For example,for the“Niao Shen Niao Fei”movement,the main focus is on the amplitude of the hands up and the amplitude of the single foot when lifted backward;for the“Wei Han Tian Zhu”movement,the main focus is on the horizontal coordinates of the nose and arms.The results are shown in Fig.5.

▲Figure 4.Structure of pose sequence classification

3.3 Algorithms of BMR System

Ancient Chinese five-tone music is divided into five keys:Gong,Shang,Jue,Zhi,and Yu.However,because of the wide variety of music libraries currently available,it is not practical to manually classify songs.Therefore,we use LSTM[15]as a backbone and design an algorithm to implement an automatic judgment of the main key of the song.The algorithm is divided into three specific steps:the main melody transcription to determine the pitch of the ending,the tonality detection to determine the key number,and the main tone discrimination according to music theory.

Building a five-tone music database is highly complex due to the different meanings of traditional five-tone Chinese tuning and western music systems.Currently,there is a lack of a relatively complete dataset on the internet.We invite Dr.XIA from Sichuan Conservatory of Music to set up a working group to help us annotate traditional five-tone music and build a small dataset.The dataset we build is used to train the network model to achieve the automatic classification of traditional Chinese music.Finally,a background music library of 500 songs is constructed to assist patients in the practice of TCM Daoyin.

▲Figure 5.Results of posture recognition and correction(PRC)system

3.4 Algorithms of PSMSystem

The detection of physiological signals is often used as a criterion for the physician’s judgment during the patient’s rehabilitation process.Therefore,our system needs to enable realtime measurement and monitoring during the patient’s TCM Daoyin training.Our algorithm is developed to design a portable and miniaturized physiological signal monitoring device.ECG signals can be used to represent the electrical signals of heart activity[17]and are widely used in the detection of arrhythmia.In this paper,the BMD101 chip is selected to acquire and detect the heart rate signal of patients.At the same time,abnormal respiratory rates are sometimes one of the best independent predictors of cardiac arrest in clinical practice.The device also incorporates an algorithm to extract respiratory signals from ECG signals to monitor the patient’s respiratory rates.

3.4.1 Heart Rate Detection and Judgment

The BMD101 chip[18]is a miniature device developed by NeuroSky specifically for bio-signal detection and processing.The device uses a single-lead detection method,which allows detection of ECG signals at theμV to mV level by simply placing the electrode pad on the chest.Moreover,it can realize signal pre-processing and calculate indexes such as mean heart rate and heart rate variability.

The chip’s built-in algorithm parses the patient’s current static heart rate from the data stream and calculates the average heart rate.When we get the patient’s static heart rate,we can get the patient’s current theoretical maximum heart rate and theoretical minimum heart rate based on the physician’s clinical studies.And based on this,we can make a judgment about the patient’s heart rate stability.

where HRmaxand HRminindicate the theoretical maximum and minimum number of heart rates per minute,respectively.And HRstaticindicates the number of resting heart rate per minute.

3.4.2 Deriving Respiratory Rates from ECG Signals

Previous studies have shown that respiratory rates can be derived from ECG signals by analyzing either the variability in R–R intervals during a respiratory cycle or the change in QRS(the Q,R,and S wave groups)amplitude during breathing.The EDR[19–20]is obtained using both the heart rate variability(HRV)method and the peak amplitude variation(PAV)method,as shown in Fig.6.

Because the actual measurement to obtain the ECG signal is influenced by body temperature changes,industrial frequency interference,and visceral activity,it generates baseline drift and other noise.We obtain the original ECG signal by removing the baseline drift from the detected R-peaks using the cubic-spline interpolation method.And due to the low frequency of the respiratory signal(0.1–0.7 Hz),we need to downsample and smoothly filter the obtained signal to finally obtain the respiratory signal.Table 2 shows the results.

4 Device Integration

▲Figure 6.ECG derived respiration(EDR)algorithm framework

▼Table 2.Results of physiological signal monitoring(PSM)system

We integrate the subsystems into a mini-PC to make the system portable and operational and to ensure that patients can practice wireless self-rehabilitation at home and in the community.And the device is equipped with a Bluetooth module as well as a 5G module,which facilitates data transmission from sensors and ensures that doctors can monitor patients’rehabilitation training in real time on a remote server.

As shown in Fig.7,we choose a Core i7-7820HQ processor and a GTX1650 graphics card with 4 GB of video memory to support the hardware requirements for our algorithm implementation.It achieves real-time feedback on the scoring of each index during patients’TCM Daoyin training.To improve the accuracy rate,we choose a high-definition camera to satisfy the needs of computer vision.Meanwhile,to collect the ECG signal from the patient,we embed a Bluetooth module in the device to enable remote wireless transmission of ECG signals.To maintain the overall portability of the device,we choose a removable power supply.The device also needs to ensure that the server receives and feeds back quickly and doctors could monitor and guide patients in time.Therefore,we embed a 5G communication module in our device.

When the equipment works,we set up a variety of options to provide a more comprehensive rehabilitation strategy for users with different needs.The specific process is shown in Fig.8.

▲Figure 7.System hardware equipment

5 Conclusions

In this paper,we innovatively combine AI and TCM Daoyin for rehabilitation assessment and training of patients with DJD.We integrate the system into a mini PC to ensure the portability of our device.The system is equipped with a Bluetooth module and a 5G communication module,and therefore fast and wireless data transmission between the system and the server is achieved.Each of our subsystems has its innovative points.The SMA system detects the patient’s posture by computer vision algorithm and gives a skeleton model to assess the patient’s spine status by our innovatively proposed scoring formulas.The PRC system classifies the posture of patients during TCM Daoyin training and proposes different scoring methods based on different standard movements.What’s more,we construct a small sample dataset.The BMR system is based on the five-tone theory of TCM to assist in patient therapy.We also build a small sample dataset to achieve the automatic classification of the five-tone music.The PSM system calculates the real-time heart rate of the patient by collecting the ECG signal of the patient.The real-time respiratory rate of the patient is separated from the ECG signal by the EDR algorithm,which achieves timely alerting of abnormal situations.

In all,our system is geared toward patients with DJD who are in remission.The system combines multiple algorithms of AI,and more importantly,it is wireless and portable so that it achieves the rehabilitation treatment of patients at home and in the community.The device has high practicality,a wide range of applications,and effective dissemination of TCM theory.

▲Figure 8.System working process