Implementation of Machine Learning and Deep Learning Algorithms with Dimensionality Reduction Methods for Internet of Things Gait Analysis and Monitoring Systems

In this paper, we present an end-to-end monitoring system, which is used for patients who have foot or ankle impairments. This system has been created to help orthopedic doctors optimize treatment for patients recovering from foot and ankle injuries. The system consists of three main parts: a wearable controlled ankle motion (CAM) boot equipped with inertial and load sensors, a web application that provides visual feedback obtained from sensors, and the implementation of machine learning and deep learning to analyze walking activity and gait. Sensors used on the CAM boot include an accelerometer, a gyroscope, and load cells. Values from sensors attached to the CAM boot are sent wirelessly to the database. The web application takes sensor values from the database and returns visual feedback on the patient’s walking patterns in the form of different graphs. The graphs can be used to analyze and determine abnormalities in the patient’s gait and serve as a visual aid for patients during rehabilitation. Sensor values obtained from the database are used to train machine learning and deep learning models to recognize and differentiate between seven activities performed by the patient. We study and compare three dimensionality reduction methods and six classifiers. As a result, we find that the joint incorporation of the dimensionality reduction method of sparse principal component analysis (PCA) and the classifier random forest (


Introduction
Research on IoT-oriented gait analysis and activity recognition monitoring systems has increased due to the widespread usage of wearable smart devices and sensors. (1)(2)(3)(4) This has enabled the collection of human activity data such as gait. Gait can be defined as the pattern observed from limb movements occurring during movement from one place to another. Various sensors such as inertial measurement unit (IMU) sensors and piezoelectric sensors can be used to observe, monitor, and distinguish between normal and abnormal gait, especially after sustaining injuries to the foot or ankle. (5)(6)(7) There have been many contributions in the form of smart shoes with attached sensors to analyze the gait of those with foot or ankle impairments. (8,9) For example, wireless joint ankle sensors can be used to measure joint rotations in three dimensions, (10) and pressure sensors can be attached to running shoes to measure the ground force applied by each foot. (11) There have been many attempts to classify, predict, and recognize various human activities by using various deep learning architectures and machine learning methods for both impaired and nonimpaired gait. (12)(13)(14) Common causes of foot and ankle injuries include falls, vehicle accidents, and sports. (15,16) Such injuries are repeatedly experienced by many and could lead to temporary, long-term, or even permanent impairments depending on the severity of the patient's condition and how effectively the injury was treated. In cases where there are abnormalities in a patient's gait or walking pattern, it is important to identify these abnormalities and treat them in their earliest stages. Injuries that are not effectively treated could result in negative long-term effects or delayed recovery.
When consulting a doctor for foot and ankle injuries, patients are often given a cast or, in most cases, a controlled ankle motion (CAM) boot to wear and are instructed to walk in a certain way to ensure a quick recovery. For instance, a patient could be asked not to apply more than a certain weight on the injured foot or to fix certain irregularities in their gait cycle. Recommendations and instructions are often given to patients as auditory feedback. This process is mostly done through physical examination, which can often leave room for errors or failure to detect certain gait abnormalities or disorders as not all significant details can be captured through observation alone. (17) Additionally, auditory feedback can be difficult for patients to understand and visualize as they do not have visual feedback to assist them. Furthermore, doctors are unable to monitor the patient's progress after they have left the hospital. These problems could hinder the recovery of the patient and may even lead to permanent damage if the patient is unable to consistently follow recommendations or instructions given by the doctor.
In this study, we have built an end-to-end IoT system with the implementation of machine learning and deep learning algorithms to help combat and solve the problems mentioned above. The system will be used to analyze, monitor, and detect abnormalities in a person's gait. The system consists of three major components: the hardware design, the web application, and the implementation of machine learning and deep learning algorithms. The hardware design includes a CAM boot equipped with sensors that wirelessly transmit data values to the database. The web application retrieves sensor values from the database and outputs visual feedback of the patient's walking activity, allowing both patients and doctors to monitor the patient's gait in realtime through a web application. Data obtained from the sensors is used to train machine learning and deep learning models to determine the patient's gait or activity. We compare different classifiers and dimensionality reduction methods to find which give the best result.
The paper is structured as follows. Section 2 describes the system that was implemented. Section 3 explains gait analysis and shows graphs of different walking patterns from patients. Section 4 illustrates the machine learning and deep learning methodology. Section 5 discusses our experimental results. Finally, Sect. 6 presents our conclusions.

Proposed System
The proposed system gives insight into a patient's walking pattern and activity after an injury to the foot or ankle. The system can detect and differentiate between different types of gait patterns. The system consists of three main parts: the hardware, the web application, and the implementation of machine learning and deep learning algorithms. The hardware, which is the CAM boot, sends data wirelessly to the database. The web application outputs visual feedback of the patient's walking pattern using sensor values from the database. Finally, data collected in the database is used to train the machine learning and deep learning models. An overview of the system can be seen in Fig. 1.

Hardware implementation
The hardware consists of a CAM boot, an ESPino32 microcontroller, an IMU sensor, a load amplifier, and four load cells. The finished hardware can be seen in Fig. 2. The CAM boot is an adjustable device, which can be used with mild to severe leg, foot, and ankle injuries and is usually used to limit foot or ankle movement. (18) Consequently, a smaller difference in the  orientation of the foot will be noticed while walking when wearing the boot. Here, the CAM boot was used as a platform to mount the sensors and collect the data. The sensors and microcontroller are described as follows.
The microcontroller used is the ESPino32 board produced by ThaiEasyElec. It is a low-cost and locally produced microcontroller that includes a Wi-Fi module and an ESP-WROOM-32 Bluetooth module. The Wi-Fi module has a range of 1 km, which allows the wireless transmission of sensor values to a database.
The IMU sensor contains both a gyroscope and an accelerometer. Here, a three-axis accelerometer and three-axis gyroscope are used. The IMU sensor used is the MPU6050 module, which employs an I2C interface. The accelerometer collects acceleration values of the injured foot, and the gyroscope collects angular velocity values. The IMU sensor can assist doctors in observing the patient's gait from patterns of the injured foot's angular velocity and acceleration. The raw data of the acceleration is configured to output values in the unit of g (the standard acceleration due to gravity). Then, the acceleration values are multiplied by 9.8 to obtain acceleration in m/s 2 . On the other hand, angular velocity values are given in the unit of degrees per second.
The load cells and load amplifier are used to help monitor the weight exerted by the patient on the injured foot. Four load cells are used, which are attached to the insole of the CAM boot. The four load cells are connected to an HX711 load amplifier. The load amplifier combines four load sensors into a standard four-wire Wheatstone bridge configuration, which outputs the total weight value from all four sensors combined when weight is applied. Finally, the circuit is powered by a power bank attached to the CAM boot.

Web application implementation
Patients and doctors can log into the web application to track and monitor the progress of recovery and view the walking activity of the patient's injured foot. After data are sent wirelessly to the database from the CAM boot, the data are displayed on the dashboard of the web application for patients and doctors to see. Information displayed includes acceleration graphs, load graphs, and angular velocity graphs, which can help give insight on the patient's gait such as the speed at which the patient is walking, how much weight is exerted, and the number of steps per second. The front end of the web application was created using HTML and CSS. The back end was created with PHP, and a Firebase real-time database was used, which is a cloud-hosted NoSQL database. With Firebase, data can be stored and synced between users in real-time. The user interface of the web application is shown in Fig. 3.

Gait Analysis
In this section, we compare and observe different types of gait. We tested our system on three different patients in three different scenarios: walking heel first, walking foot flat, and walking toe first. Each scenario will be analyzed in detail in the following sections using the dashboard from the web application we have created. The different types of gait will be described on the basis of patterns of exerted weight, acceleration, and angular velocity in the x, y, and z axes. The scenario of walking heel first is that of a normal gait, while the other two scenarios are instances of abnormal gait. When walking heel first, the participant strikes their heel to the ground first at the beginning of each gait cycle. To first understand and recognize abnormal gait, we study the normal gait cycle, which is shown in Fig. 4. There are four commonly observed events in the gait cycle: heel strike (HS), foot flat (FF), heel-off (HO), and toe-off (TO). (19,20) HS is when the heel strikes the ground, FF is when the foot is placed flat on the ground, during which the most weight is applied, HO is when the heel leaves the ground, and TO is when the toe leaves the ground.

Scenario 1: Walking heel first
Walking heel first is considered a normal gait. Walking heel first involves striking the heel on the ground first at the beginning of each gait cycle. The line graphs for exerted weight, acceleration, and angular velocity are shown in Figs. 5(a)-5(c), respectively. Each graph displays a total of 100 records, corresponding to a walking time of 5 s (oldest record to newest record).   From these graphs, we can analyze the participant's gait in terms of cadence, walking speed, and how much weight the patient exerts on the injured foot. From the weight graph, we can see that the patient takes 1.2 steps per second and exerts a weight of around 10 kg in each step. We can spot the four main stages of the gait cycle clearly in the weight graph. Each stage is highlighted in Fig. 6.
Following our gait cycle diagram in Fig. 4, we see that the gait cycle begins with the striking of one's foot on the ground. We can spot this stage (HS) by a small increase in the exerted weight graph (marked by number 1 in Fig. 6) before the weight reaches its peak. This is due to the increased pressure on the heel but not yet the whole foot. The next stage detected after the heel strike in the graph is the FF (number 2 in Fig. 6). In this stage, the most weight is applied on the foot, which explains the significant increase in the exerted weight as can be seen from the graph. After that, the participant lifts the heel off from the ground to take the next step. This decreases the weight on the foot as we can see from the drop in the graph of the exerted weight (number 3). In the last stage, the toe leaves the ground (TO) to prepare for the next HS. We can also detect these gait events in the acceleration and angular velocity graphs. It is noticeable that during the FF stage, there is no or very little change in the acceleration and angular velocity in all axes. It is also important to point out that the angular velocity values in the z axis are always near zero as the orientation in this direction is limited due to the CAM boot. This trend can be observed in walking FF, walking toe first, and walking heel first.

Scenario 2: Walking FF
In the second scenario, the participant walks FF. This means that the participant does not strike their heel on the ground first and transit to the FF stage, but instead, they instantly place From Fig. 8, we see three main detectable gait events: lifting foot motion (marked by number 1), foot falling motion (number 2), and FF (number 3). Unlike the load cell graph for walking heel first, we see that there is no gradient change in the exerted weight as there is no HS motion in walking FF. The participant skips the stage HS, which is the stage where the heel cushions some of the weight, and instantly places their foot flat on the ground. As a result, at each step, the increase in exerted weight is abrupt, with the value increasing from almost zero to 10 kg immediately.
From the acceleration and angular velocity graphs, we see that there is little or no change in acceleration or angular velocity during the FF stage, which was also the case in the previous scenario. In addition, it can be seen that a significant positive increase in acceleration and angular velocity values occurs when the participant performs the lifting foot motion. Similarly, there is a significant negative increase in acceleration and angular velocity values when the participant performs the foot falling motion.

Scenario 3: Walking toe first
In this scenario, the participant walks toe first, where instead of striking the heel onto the ground first at the beginning of each gait cycle, the toe is used instead. Participants with this gait only exert a small amount of weight on the toes of the foot of interest and nowhere else. The exerted weight, acceleration, and angular velocity graphs are shown in Figs. 9(a)-9(c), respectively. We can see that the steps taken are more random and that little force is applied at each step. We also notice there is little change in the acceleration and angular velocity values.

Machine Learning and Deep Learning Implementation Methodology
We compare multiple machine learning algorithms using dimensionality reduction methods and deep learning neural networks. The diagram in Fig. 10 illustrates our methodology to investigate and compare different algorithms. We first load the data signals, which consist of the load, acceleration, and angular velocity in the x, y, and z axes. Then, we segment the data into frames of 4 s and encode labels in the segmented data.
To compare different machine learning classifiers and dimensionality reduction methods, we extract features from our data. A pipeline is constructed containing a standard scaler, a dimensionality reduction method, and a classifier. It is used to prevent data from leaking between training and test data points. The data that pass through the pipeline are cross-validated via stratified K-fold cross-validation. For deep learning, data are split into training and test datasets. The data are then scaled and trained through the deep learning architecture. Further information on each stage will be given in the following subsections.

Data collection
Our dataset consists of six people, referred to as participants A-F. Details of each participant are listed in Table 1. The data recorded comprise x-axis, y-axis, and z-axis acceleration signals, x-axis, y-axis, and z-axis angular velocity signals, and load (exerted weight on foot). The collected data are stored as JSON files.
We collect the data in seven scenarios: sitting position, standing position, walking upstairs, walking downstairs, walking foot first, walking toe first, and walking heel first. These activities have been suggested by a doctor to be the basic activities carried out by a person in a day. Table  2 shows the number of records collected for each activity. Each participant in the dataset is instructed to perform each activity repeatedly for an amount of time.

Data segmentation
The results of a previous study conducted to determine a suitable sampling rate suggested a sampling rate of 22 Hz. (21) As a result, our data are collected at a frequency of 22 Hz. The data are segmented into frames of 4 s each with an overlapping rate of 50%, following suggestions regarding the window size and sliding window techniques made in Refs. 22 and 23. Therefore, the frame size is equal to 88 records. One frame will only consist of records belonging to one activity. We iterate over each JSON data file to create frames. Table 3 shows the number of frames extracted for each activity.

Data scaling
StandardScaler is used to reduce large differences between points. This helps lessen the impact on the model training. Not scaling the data may lead to skewed results when training the model. StandardScaler transforms the data to make the mean of the distribution of each feature equal to 0 and the standard deviation equal to 1. We use the sci-kit learn library to scale our data.

Dimensionality reduction approaches
When training with a large number of features, models may be negatively affected by the curse of dimensionality and poor generalization. (24) Dimensionality reduction approaches have been shown to improve the performance of machine learning models. (25) In this study, three dimensionality reduction approaches are used: principal component analysis (PCA), Kernel PCA (KPCA), and Sparse PCA.
PCA is a dimensionality reduction method that converts a set of features from a higher dimension into a lower dimension. It achieves this by maximizing the variance in the new subspace by selecting the fittest principal components. (26) PCA is a linear algorithm. Kernel PCA extends PCA by applying a kernel to the dataset to improve the performance for features that are not linearly separable. (27) Sparse PCA extends PCA differently. PCA uses linear combinations of all of the original features, producing new features that are difficult to interpret. Sparse PCA solves this problem by using linear combinations of a few features. (28) The libraries used to perform dimensionality reduction are listed in Table 4.

Machine learning classification using cross-validation
Various classifiers have been used and tested in previous studies. (29,30) In Ref. 31, it was shown that ensemble methods can improve the classification performance. Therefore, we compare both simple and ensemble classifiers in this study. We test the following machine learning classifiers on our dataset: K-nearest neighbor (KNN), decision tree (DT), support vector machine (SVM), random forest (RF), XGBoost (XGB), and LightGBM. To evaluate and compare each classifier, we apply Stratified K-fold cross-validation from the sci-kit learn library, where K = 10. The libraries used for the learning classifiers are listed in Table 5.

Deep learning architecture
Recently, there have been many approaches where neural networks are used to classify and differentiate between gait or human activities. (32,33) In this study, we also implement a simple convolutional neural network for evaluation and comparison. It consists of 2D convolution layers, a dropout layer, and dense layers. All layers use the ReLU activation function except for the last layer, which uses SoftMax. The Adam optimizer is utilized with the default parameters. The deep learning architecture is shown in Fig. 11.

Experimental Results and Discussion
Previously, we described the process by which we investigate and compare six machine learning classifiers (KNN, DT, SVM, RF, XGB, and LightGBM) and a simple convolutional neural network. Several dimensionality reduction approaches are applied to each machine learning classifier to evaluate and determine the best combination. The default hyperparameters are used for a fair comparison. The default hyperparameters of each classifier are listed in Table 6 for reference. The performance results are measured on the basis of accuracy, F1, precision, and recall scores.

Results of machine learning classifiers with dimensionality reduction methods
We use stratified K-fold cross-validation with K = 10 and shuffle = true to evaluate the performance of our models. This evaluation method splits a dataset into K sets, where each set is trained K − 1 times and tested once. Each set conserves the same ratio of samples of each class. The stratified K-fold process can be seen in Fig. 10. In Table 7, we show the classification performance (accuracy, precision, recall, and F1 scores) of the six classifiers (KNN, DT, SVM,  XGBoost and LightGBM use gradient boosting. Here, trees are continuously added to the initially built tree to cover the weakness of the previous model until it reaches the set amount specified in the hyperparameter called n_estimators. This is why ensemble methods generally have superior performance. We also notice that PCA gives the lowest scores when incorporated with other machine learning classifiers, while Sparse PCA significantly improves the performance of the classifiers, with the three highest performances obtained with Sparse PCA. Since we know that Sparse PCA utilizes linear combinations of only a selected set of features from the dataset, this could mean that there are features that are unnecessary or have a negative impact on the dataset. Fig. 12 shows the importance of each feature decided from the combination of RF and Sparse PCA from highest importance to lowest importance. We can see that many features are considered unnecessary.

Deep learning results
The deep learning model is constructed using TensorFlow 2 following the architecture in Fig.  11. For deep learning, data are split into training and test datasets with 80% being the train dataset and 20% being the test dataset. A small model is chosen to minimize the running time

Comparison with previous studies
In this section, we compare the results obtained in this study with those in the literature. Similar to in previous studies, we showed that RF outperforms other classifiers. In Ref. 34, in which the dimensionality reduction methods of PCA and linear discriminant analysis were compared, the highest accuracy of 87.5% was obtained using RF and PCA, with acceleration in the x, y, and z axes used as the only features. In Ref. 35, different classifiers were tested against three different datasets. The highest accuracy scores were again obtained from RF and ranged from 96.74 to 100%. In Ref. 36, CNN and LSTM were applied to the UCI HAR dataset and the highest accuracy obtained was 92.13%. It is important to note that different datasets were used in each study and that the results obtained in this paper are based on an original dataset collected specifically for this study.

Conclusions
We have created a system that can be used to monitor and analyze the foot pressure and movement of a person with foot or ankle impairments. The system will help orthopedic doctors optimize treatments for patients. The system consists of a CAM boot with attached sensors, a web application, and the implementation of machine learning and deep learning algorithms, which help determine and differentiate between normal and abnormal gait. We have investigated and compared the classification performance of machine learning classifiers with incorporated dimensionality reduction methods. Our experimental results have proven that the RF classifier with Sparse PCA incorporated outperforms all other methods. As future work, we would like to explore more complex deep learning architectures and expand our activity dataset.