Unmanned Mobile Multipurpose Monitoring System—iMonitor

temperatures at a border control for quarantine inspections during the outbreak of contagious diseases. Furthermore, the system is equipped with four gas sensors to detect up to 12 different kinds of gaseous substances or particulate matter so that the user can be aware of the environmental quality around the self-balancing car. The outcome of this work can be applied to public health monitoring, dynamic crowd monitoring at airports and harbors, and the monitoring of hazardous or disaster-affected locations for search and rescue to reduce the costs associated with manual monitoring and the risk of exposure to hazards.


Introduction
Monitoring systems have been widely used in different areas, such as sewerage systems and underground holding tanks, where methane gas may build up and cause danger to inspection personnel. To avoid methane poisoning, sewerage systems may first be inspected by robots. (1) In hazardous areas or disaster-affected locations, autonomous monitoring vehicles may be used in surveys to improve the efficiency and reduce the risk. (2) In industry, various sensors installed in factories aid the detection and control of production lines. A thermographic system can also rapidly detect the temperature of components and obtain necessary information. Our goal is to integrate these functions into a multipurpose monitoring system, as shown in Fig. 1. The system is mounted on an unmanned vehicle and various monitoring devices are integrated to achieve unsupervised detection. The proposed system with multiple monitoring functions can be used in different areas. The system provides a mobile application for real-time monitoring and manipulation. (3) The proposed multipurpose monitoring device is mounted on a small, two-wheel, self-balancing car so that the system remains stable while in motion, even for uphill climbing. A micro video camera and a thermographic camera are mounted on the self-balancing car for video recording. The recorded images undergo facial tracking and recognition (4) and forehead temperature detection at the terminal, and a convolutional neural network (CNN) (5) is used to determine if the face belongs to a stranger. The proposed system incorporates four types of gas sensors that display real-time gas or particle information on a mobile application via a Wi-Fi network. The gas sensors can detect up to 12 different gaseous substances or particulate matter, PM2.5, CO 2 , LPG, CO, smoke, NH 3 , NO 2 , C 3 H 8 , C 4 H 10 , CH 4 , H 2 , and C 2 H 5 OH, so that the user can recognize the environmental quality around the selfbalancing car. The proposed system can also be applied to the exploration of industrial-disasteraffected locations, holding tanks, or other high-risk areas.

Related Work
In Ref. 6, an Android intelligent mobile device was used as a remote interface that was connected over a Wi-Fi network to accomplish real-time video transmission to the mobile device and the control of an autonomous vehicle. In Ref. 7, a streamed video was successfully viewed in a mobile application in real time over a Wi-Fi network, and a remotely controlled vehicle's hardware was also upgraded to include a pulse width modulator, cross-country ability, and the ability to explore harsh environments. In Ref. 8, a vehicle was remotely controlled by a mobile device that was connected to a camera mounted on the vehicle and monitored the captured video. Simple robotic arms were added to the vehicle to provide basic assistance and rescue functions.
Facial recognition systems have improved alongside the development of artificial intelligence (AI). Many researchers have begun to investigate related research topics. For example, in Ref. 9, dynamic facial recognition of users was performed for an access control system with deep learning techniques to determine if the user was a member of the laboratory. In Ref. 10, Python and OpenCV were combined to achieve image processing for facial recognition based on supervised learning classification. Most related research applies facial recognition systems to access control systems; however, this may not be suitable for certain open environments such as airports or schools. Therefore, the system can apply a facial recognition system in open areas for effective environmental monitoring. The system can also be used for search and rescue in disaster-affected areas or the surveillance of hazardous locations.
In recent years, research and development in cloud systems have attracted considerable attention. Cloud computing can reduce the load at the terminal CPU and centralize computations, enabling users and cloud servers to perform data access, storage, and settings with ease. In Ref. 11, sensors were used to collect temperature, humidity, and gas (such as carbon monoxide, methane, and natural gas) data, as well as to perform real-time analysis of the data and store the data in a cloud database. The system also allowed the user to remotely control an autonomous vehicle through a mobile application and monitor the situation at home through video cameras. We have referenced this work and improved the system by adding monitoring, thermography, image capturing, and video recording functions to the remotely controlled autonomous vehicle. As a facility of the facial recognition system, if suspicious persons appear at the monitored site, they will be tagged with a red rectangle.

Integration of Internet of Things (IoT), cloud platform, and mobile application
In this work, we have adopted the concept of the IoT, displaying the streamed video captured by a micro video camera mounted on a self-balancing car or thermal images taken by a thermography module in real time on a mobile device via a Wi-Fi/4G wireless network, (12) as shown by arrow 3 in Fig. 2.
The proposed system utilizes the open-source software PhpMyadmin (13) as the backbone of the cloud computing service platform. (14) The user can use the mobile application to control the self-balancing car directly via the cloud platform. When the user presses the directional buttons on the mobile application, the corresponding values are uploaded to the cloud platform. The self-balancing car constantly reads the data from the cloud platform and executes the associated actions. By uploading the data as updates, the data read by the self-balancing car always correspond to the most recent buttons pressed by the user, as shown by arrow 1 in Fig. 2.
The videos or images captured by the mobile application are uploaded to the cloud storage in real time without taking up memory space within the embedded platform on the self-balancing car. In this manner, the videos can be recorded continuously throughout the day without running out of memory or overwriting previously recorded data. The four gas sensors mounted on the self-balancing car can detect up to 12 different types of gaseous substances or particulate matter, for example, CO, CO 2 , PM2.5, and liquefied petroleum gas, and update the information on the cloud platform. The user can be instantly aware of the air quality surrounding the selfbalancing car through the mobile application, as shown by arrow 2 in Fig. 2.

Unmanned mobile multipurpose monitoring self-balancing car
Figure 3(a) shows the actual structure of the unmanned mobile multipurpose monitoring system. The system uses a two-wheel, self-balancing car as a carrier vehicle, which remains stable even during uphill climbing. The balancing approach is to continuously detect the vehicle's tilt and speed during movement and use the values to cancel the vehicle's tilt by accelerating the wheels. The concept is shown in Fig. 3(b). (15) Suppose an external force causes the vehicle to undergo angular acceleration ω(t) along the direction perpendicular to the base of the vehicle, then the motion equation for the vehicle's tilt, the wheel acceleration a(t), and the acceleration caused by the external force, ω(t), are given by Eq. (1). When the tilt θ is small, the sine and cosine components may be ignored and the simplified motion equation is shown in Eq. (2). When the vehicle is still, the wheel acceleration is a(t) = 0, and the motion equation can be further simplified to Eq. (3). The equations can be used for three different situations and effectively improve the balancing efficiency.
The Raspberry Pi 3 Mode B platform was used as the main operating system for the hardware development of the unmanned mobile multipurpose monitoring system, as shown in Fig. 4(a). The system has a 1.4 GHz quad-core CPU. The CPU is compact with Bluetooth and Wi-Fi network functions and operates under the Linux operating system, making it suitable for processing the switching between multiple tasks.
To import AI deep learning technology into edge computing, we added the Vision Bonnet extension to the Raspberry Pi 3 Mode B platform, as shown in Fig. 4(b). Vision Bonnet is an extension for Raspberry Pi and includes a microprocessor with machine learning functions that can be connected to Raspberry Pi to process the videos captured by the micro video camera (Raspberry Pi Camera). In this manner, the deep learning computations can be directly executed on the terminal device, which acts as an intelligent monitor, (16) as shown in Fig. 4(c), where the connection between Vision Bonnet and Raspberry Pi is on the right-hand side of the figure.
In addition, the system is also equipped with an infrared thermographic camera module using the MLX90640 far-infrared thermal sensor array, which can detect the infrared radiation energy distribution of objects in the camera field of view. After quantization, the surface temperature distribution of the object in the camera field of view can be obtained to generate a thermal image, as shown in Fig. 4(d). Accordingly, the module can be combined with face recognition at the same time. When the system detects a face, it will automatically detect the forehead temperature through the thermographic camera and display it synchronously in the real-time image streaming, as shown in Fig. 4(c). The infrared thermographic camera is on the left-hand side of the figure and users can switch to display the video stream or the infrared thermal image based on their requirements.
Owing to the limitations of the Raspberry Pi, the images captured by the camera cannot be stored into the camera mentioned above that provides a stream as a video for facial recognition. Therefore, we can install one more microcamera to enable pictures and videos to be taken at the same time. This camera, shown in Fig. 4(e), uses the smaller Raspberry Pi Zero W to reduce the burden on the system.
The system has four gas sensors to achieve multipurpose monitoring. Real-time gas information can be displayed on the user's mobile application so that the user is aware of the environment quality around the self-balancing car. The system can detect up to 12 various gaseous substances or particulate matter: PM2.5, CO 2 , LPG, CO, smoke, NH 3 , NO 2 , C 3 H 8 , C 4 H 10 , CH 4 , H 2 , and C 2 H 5 OH. The gas sensors are shown in Figs. 4(f)-4(i).

Facial recognition system and forehead temperature detection
AI deep learning techniques are often computationally expensive (17) and may require dedicated server rooms to perform the computations. However, the large amount of computations is unsuitable for the multipurpose monitoring system with real-time performance requirements. If intelligent image recognition is performed after acquiring the streamed video via the Wi-Fi network, (18) the real-time performance may be compromised owing to problems such as the availability of bandwidth. Therefore, we adapt a terminal AI facial recognition system (19) to the self-balancing car and shift part of the AI computations from the cloud platform to the terminal, which is the micro video camera in this case, to perform recognition analysis and transfer the streamed video to the mobile application via a Wi-Fi network, so that the AI facial recognition system can be part of the edge computing field category. (20) The terminal AI facial recognition system performs deep learning computations directly on the video camera. In the proposed system, we use the library from the OpenCV image processing system (21) and implement the AI facial recognition program in Python. The program has three parts: (1) face detection (22) and data acquisition, (2) CNN model training, and (3) CNN facial recognition.
We use the Haar cascade classifier (23) to perform target detection of the face. The particular machine learning method uses a large number of images with and without faces to train the cascade classifier functions and extracts the features to detect the targets in other images. When a human face is detected, the system records the coordinates of its position, measures the temperature at the coordinates through the thermographic camera, that is, the forehead temperature, and finally returns the value. An area of interest (24) is marked by a red rectangle to indicate the detection of a face, and its forehead temperature is also displayed, as shown in Fig. 5(a).
The forehead temperature measured by the thermographic camera will be different from that measured by the infrared forehead thermometer. Therefore, the forehead temperature measured by the thermographic camera must be calibrated to match that measured by the infrared forehead thermometer. First, we measure the forehead temperature ij tc t n times with a thermographic camera and find the average value. This is done m times, and then the overall average value of the forehead temperature measured by the thermographic camera, tc t , is found using Eq. (4). At the same time, the forehead temperature ij f t is measured p times with the infrared forehead thermometer and the average value f t is calculated using Eq. (5), then the difference in the forehead temperature between the thermographic camera and the infrared forehead thermometer, tc t ∆ , is calculated using Eq. (6). Finally, we measure the forehead temperature l tc t q times with the thermographic camera to find the average value, then the compensation of the deviation of the forehead temperature is obtained using Eq. (7). The forehead temperature measured by the thermographic camera, ˆt c t , can then be estimated. Through the above method of calibrating the measured forehead temperature, the accuracy of the forehead temperature measurement can be effectively improved. A dataset is created to store the registered face data. Since the facial recognition process compares and analyzes the captured image data and the existing image data, face image data need to be obtained first. We store each face ID along with a set of gray-scale face detection images. Each set of IDs is associated with around 30 captured image samples, as shown in Fig. 5(b).
Once the user has performed the face registration process, we extract all user data from the dataset and use the Keras CNN to build a model, as shown in Fig. 5(c). During the training of the model, the CNN automatically extracts features from the training data, constructs a fully connected neural network to implement the face classifier, (25) and stores it as a trainer.yml file, as shown in Fig. 5(d). When the micro video camera captures a face, the detector returns the estimated ID, index, detected forehead temperature, and the degree of confidence in the estimated result if the person's face has been captured and trained before, as shown in Fig. 5(e).

Unmanned mobile multipurpose monitoring self-balancing car
The proposed system uses a self-balancing car as the carrier vehicle, which can advance steadily even on a slope. The mechanisms of the self-balancing car and mobile multipurpose monitoring system are respectively shown in Figs. 6(a) and 6(b). We use a TB6612FNG motor driver module to drive a GB37 direct-current, geared motor. An MPU6050 gyroscope and a speed-measuring motor (composed of a motor and a Hall effect sensor) are respectively used to obtain the tilt angle and the distance data of the vehicle. The balance output is calculated using formulas implemented on an Arduino development board, which is used to control the wheel's acceleration to cancel the tilt of the self-balancing car. Through the above-mentioned process, the vehicle can maintain its balance while in motion. (26) Figures 7(a) and 7(b) respectively show the actual operation of the self-balancing car moving on uphill and downhill sections while maintaining its balance and moving forward steadily. The technological background (27)(28)(29) for  the iMonitor encompasses the following areas: IoT, mobile application, database system, AI facial recognition, face tracking, gas detection, and infrared thermography.

Live streaming, image capturing, and video recording
The system remotely controls the self-balancing car through an intelligent mobile device. The self-balancing car automatically detects the slope angle and speed to control the wheel's acceleration, which in turn is used to adjust the tilt angle for balance so that the vehicle remains stable even when going uphill. A micro video camera is also mounted on the vehicle to capture and display images on the mobile device in real time through a Wi-Fi network. The user can hold down the Streaming button on the mobile application's start screen to view a streamed video, which shows the streamed video taken by the micro video camera on the self-balancing car, as shown in Fig. 8.
The system also includes image capturing and video recording functions. When the user presses the Photo button or the Recording button, the system automatically uploads the captured images to the cloud storage without taking up internal memory space on the mobile device. The user can also press the Folder button on the mobile application's start screen to connect to the cloud storage to examine the stored images or videos and monitor the environment, as shown in Figs. 9(a) and 9(b).

Thermal images
In addition to the streamed video for facial recognition, the user can also switch to thermal images at any time. In the event that some environmental factors affect the viewing quality of the streamed video, the user can still operate the system by examining the thermal images. The system can even be used in military, industrial, or medical applications. For example, as a countermeasure in the recent outbreak of coronavirus, the proposed system may be utilized at airports, harbors, and other border control points as a mobile patrol to detect affected persons with a high temperature via thermal imaging. The system can also display the temperature at the lower right corner of the mobile application to increase the detection speed, as shown in Figs. 10(a) and 10(b). In industrial applications, thermal imaging can rapidly detect abnormally high temperatures of components or machines during processing and diagnose possible malfunctions as shown in Fig. 10(c).

Air quality detection
The system includes four gas sensors to detect up to 12 different types of gaseous substances or particulate matter, PM2.5, CO 2 , LPG, CO, smoke, NH 3 , NO 2 , C 3 H 8 , C 4 H 10 , CH 4 , H 2 , and C 2 H 5 OH, and update the database via a Wi-Fi network so that the user is aware of the concentration of gases in the surrounding environment through the mobile application as shown in Figs. 11(a) and 11(b). With the ability to detect 12 different types of gaseous substances or particulate matter, the system can be widely applied to different areas. Examples include underground sewerage systems, where methane gas, which mainly contains CH 4 , is often generated, or the detection of harmful particulate matter such as PM2.5, or harmful gases that are often generated in factories such as CO 2 , NO 2 , and NH 3 , as well as flammable gases such as CH 4 , LPG, and C 2 H 5 OH that might be present in hazardous areas. The system successfully compared detected faces with registered faces in the database and displayed the associated ID with the current measured forehead temperature. If the person had not been registered, the system classified the face as belonging to a stranger and measured the forehead temperature. In the experiments, the system was able to accurately detect faces and display the associated IDs with the forehead temperature for both moving and stationary subjects as shown in Figs. 12(a) and 12(b).

Experiment 2
The forehead temperature was first detected using a face recognition system. When a face was successfully detected, its coordinates were recorded and the temperature at the coordinates was detected by the thermographic camera module. Owing to the difference in resolution between the micro camera and thermographic camera, when the system successfully detected a human face in real-time images, we marked the thermal sensor's detection range with a blue square to facilitate the user's operation. Outside of the sensing range, the human face could still be recognized, but the forehead temperature displayed "None", indicating that the system has not detected the forehead temperature, as shown in Figs. 13(a) and 13(b).

Experiment 3
In the experiments, the system required a full-face outline for successful face detection during face tracking. The system was unable to mark the face if the video camera only captured the profile of the subject, as shown in Fig. 14. If the person's head is lowered, the system may also fail to detect the person correctly, as shown in Fig. 15, in which two faces should have been detected but the system only detected the face on the right since the head of the person on the left was lowered.

Experiment 4
During facial tracking, the system could simultaneously detect a maximum of three faces, although the detection of the third face was less stable, as shown in Fig. 16. In the experiment, facial tracking was most stable when there were two people in the frame, as shown in Fig. 17.

Conclusion
We implemented a multipurpose monitoring system that is mounted on a stable, selfbalancing car and integrates a cloud platform, IoT, and mobile application to achieve an unmanned mobile monitoring system. The system can be widely utilized in industrial, medical, military, and other applications. The system is connected to a Wi-Fi network and has no restrictions on the remote-control distance. A facial recognition system and the capability of forehead temperature detection have been added to the system, as well as live-streamed videos for monitoring and the ability to rapidly detect suspicious subjects. Owing to the massive memory provided by cloud storage, images or videos can be captured continuously for future inspections. Furthermore, the infrared thermography module enables the system to be switched to monitoring through thermal images to compensate for conditions where the streamed video cannot be viewed clearly due to lighting problems. The system can even be adapted as a mobile body temperature detection unit for quarantine when there is an outbreak of a contagious disease by detecting people with abnormally high temperatures at border control locations. The four gas sensors on the system can detect up to 12 types of gaseous substances or particulate matter, informing the user of the surrounding air quality. Overall, we have achieved the goal of providing a real-time, multipurpose monitoring system.