Software engineering model based smart indoor localization system using deep-learning

ABSTRACT


INTRODUCTION
Localization systems are recently appeared to provide the information of location of different objects, such as people, animals and things.Location finding is normally acquired by using global positioning systems (GPS) for outdoor environments.Since GPS has a lack of the sight line to satellite and a lot of other obstacles, it is not working for indoor environments [1].The localization is very important for the paralyzed people, and their families as it eases the live for them and displaces their concerns.In addition, it saves time of searching for the disabled persons if they were lost for any reason.
In [2], the authors studied finger printing-based indoor localization in commodity 5-GHz Wi-Fi networks.They proposed a system called BiLoc, which uses bi-modality deep learning for localization in the indoor environment using off-the-shelf Wi-Fi devices.The experimental results validated the superior performance of BiLoc over several benchmark schemes.In [3], the authors designed an indoor localization system based on crowed sourcing, which can construct the map of WiFi radio using its data, the map of indoor location is turned to indicative map, the authors show results that their proposed network can minimize the varity problems caused by changing environment.In [4], a convolutional neural network (CNN) model for indoor localization in multi-floor buildings was produced using WiFi received signal strength values.These values have been taken from the access points in wireless LAN.The results for this model showed accuracy for training model about 100%.In [5], the authors used pervasive Wi-Fi and attractive field data for indoor limitation.Exploratory outcomes demonstrated that profound network models consolidating attractive field and Wi-Fi finger printings improved indoor localization precision.The training phase of deep positioning was computationally intensive, the testing phase was fast and suitable for real time indoor localization.In [6], the authors designed WiFi deep, which was a Wi-Fi -based indoor fingerprinting localization system that can achieve robust and high accuracy tracking in the presence of device heterogeneity.The system employed a model regularization to enable the network to generalize and avoid over-fitting, leading to a more robust and stable models.The results showed that the WI deep comes with a localization accuracy better than the state-of-the-art systems by at least 53% and 29.8% in the large and small environments respectively.In [7], they proposed a model that learns profound learning scene acknowledgment technique.This depends on limitation improvement utilizing the technique for exchange learning on inception V3 network.Model feature information was added to assist in scene recognition [8].In [9], The authors utilized an advanced mobile phone-based movement acknowledgment for indoor restriction utilizing a convolutional neural system.These exercises can be utilized as the tourist spots for indoor localization.These exercises could be utilized as the tourist spots for indoor confinement.Another convolutional neural system has been intended to become familiar with the best possible features consequently.
In this article, an indoor object localization model is proposed based on deep learning [10] and deep CNN [11].The proposed model contains two levels.First level, or software level, is used for collecting the images dataset, done using raspberry pi III camera.This camera is fixed on a simple robot of two wheels' car.The dataset is collected and prepared by gathering and pre-handling images information.It uses as input for the proposed model.The model is trained until it reaches to a satisfying accuracy for both training and validating dataset.Then the model is saved.In the second level, or can be called as hardware level, is used for gathering and controlling the electric robot car that carries on the raspberry pi III and its camera.This camera is not only used for preparing the dataset, it is also used to take a real time image when a certain condition happened or when an authorized user is asking about an object's place.In this case, the raspberry pi III captures the image and send it to the proposed inside the Raspberry Pi.Then, the proposed system waits for the response.The prediction of the model with certain accuracy is supposed to be the prediction of the place itself [5,12] and [13].

PROPOSED SOFTWARE ENGINEERING MODEL
It is well known that the provided algorithms are required to be more reliable and have all the ability of being expanding and flexible.In this work, a software engineering model is proposed as a base stricture for the utilized algorithm.This is to ensure the reliability, scalability and flexibility of the proposed algorithm.The mentioned algorithm is concepts of performing the suggested method of detecting the affected areas in the brain.Figure 1 shows the block diagram of proposed model that explains stages of method as a work flow.It is shown from Figure 1 that the system requirements are collected by the first round for preparing the structure of the designed algorithm.At the second round, the initial design of the suggested algorithm is performed based on the system requirements.Different levels of developments have been done in the third round depending on the initial design of the proposed algorithm in iteration way.The last version of the proposed algorithm is tested in the last round for ensuring the reliability, expandability and flexibility of the it.

PROPOSED SYSTEM
As mentioned earlier, the proposed system clearly contains two different phases, designed based on the proposed software engineering model.The first one, can be called as the offline phase, which contains all the programming algorithms starting with creating the dataset and processing it, ending with saving a well-trained model.The second phase, or the online phase, in which, a real time images are taken, security where localization can significantly improve security conditions the world over.Client versatility examples and communication can be utilized to distinguish conceivable dangers that may present security risks.Similarly, in war zone, the military can follow its advantages through a restriction framework that can improve the general task and increment the odds of fruitful activity.The final classification is then assigned to its estimated location.As shown in Figure 2

Proposed methodology
In this section, the dataset is collected and prepared.It also introduces the CNN model algorithm and training procedure.Moreover, this section provides the website design that ia based on the proposed algorithm.At the other side, the object localization system is introduced with the related proposed algorithms and flowcharts.The proposed overall system is started with the software part and followed by the hardware part as shown in Figure 3.The following steps describe the method used to build the model briefly: − Starting the collection of a dataset for training the model.This dataset must cover all the desired areas that is wanted to be added to the classification model.− Preparing this dataset by adding the necessary operations, such as cropping the images to be all of the same size.Moreover, adding a filliped copy of the same images for increasing their number in each class.− Designing the CNN layers of the model with Keras library.Then, the system trains the model until reaching the desired accuracy and save them in the Raspberry Pi for later use.− The software part ends with creating a simple website that contains the controlling buttons on the hardware parts.This website is built by using the HTML, PHP, CSS and python scripts.− The first step in the hardware part contains gathering and building a simple prototype of a robotic two wheels' car with its battery and motor driver.− Combining the Raspberry Pi and related accessories that include camera, LEDs, breadboard, voltage step regulator, two ultrasonic sensors.All these components are connected together and placed in a black box for arrangement.
− The Raspberry Pi is the core of controlling the hardware part.It is connected by WiFi to the website.When the search button is clicked in the web, a certain procedure is followed by moving the car and capturing real time images.This procedure is introduced in details in the next sections.− The last step is finding an appropriate classification accuracy and assigning it to its class that is related to its corresponding classified indoor location.
Figure 3.The proposed system methodology flowchart

Dataset collecting
The dataset is chosen very carefully, with notifying the angle of capturing the images and the number of images that covers the included areas that are needed to be classified.In these images, different situations and different positions are taken in consideration, as shown in Table 1.In this part, the data set collection and choosing are discussed.Moreover, as well as the preprocessing operations, such as cutting, zooming, and shearing are also introduced.The camera is located about 20 cm from ground on the robotic car with vertical angle of 15 degrees.After all datasets are collected, each class is assigned to one room.This is to reduce the complexity as it is not sensory to include all walls in the dataset because some images in the collected dataset are taken in a way that covers more than one corner inside the room.

Model training
The model is trained from scratch, because there is no pre-trained model used for initialization.The main source of data is the images available with Raspberry Pi camera.The proposed CNN with deep learning is designed to train a number of images and test others [15] and [16].Each entered image to the CNN model passes through many levels of convolution layers.Some of them contains a filters and pooling, while the others include fully connected layers.Finally, such an image passes through a soft max filter [17].The following steps illustrate the building of the CNN model: − In the first convolutional layer, 20 filters of size 5x5 pixels are applied to the input image of size 224x224, followed by a rectified linear unit (ReLU).A max pooling layer is used for taking the maximal value of 2x2 regions with strides of 2x2.− The output of the previous layer is then processed by the second convolutional layer.It contains 50 filters of a size 5x5 pixels.Again, that is followed by a ReLU and a max pooling layer.− This convolutional layer's output is passed to a fully connected layer and contains 500 neurons, followed by a ReLU.− At last, the yield of the last completely associated layer is nourished to a soft max layer that appoints a likelihood for each class of 8 classes.Then it assigns the class number to its corresponding location in one of the three rooms mentioned earlier, with a probability goes from 0 to 100%.As shown in Figure 4.
Training the CNN model is the next step, see Table 3, it takes a time ranges from 88 to 94 second for each epoch.This is because the deep learning model training requires a high ability GPU.The proposed model is trained online in Google Colab, which gives a free GPU on a free cloud service for developing applications of deep learning [18] and [19].This is done by loading the dataset to the Google drive for opening the Google Colab and making the model.Training tries are done with different number of classes from 4 to 8, in both laptop and online on Google Colab, but there is no need to mention the results of these tries.The important thing to know is that, it has been reached to a point in which only small number of epochs is enough to train this model with satisfying results.Therefore, only 5 epochs are taken each of which produces different accuracy and loss metrics for both training and validating dataset.

Website design
After the model had been trained and reached to a satisfying accuracy results, which was 99.6% for training dataset and 100% for validation dataset.This model is saved to the RPi III memory to be used in the hardware level.The saved model cannot be used directly for predictions, because it is saved as a python script.So, an interfacing mechanism must be added for the user to be able to contact with the model in some way.Figure 5 shows a flowchart for the designed website, which contains the steps that must be followed to operate this system.The following steps are described in the next lines: − A simple web page is created inside the RPi.For security reasons, this web page contains a login button at the starting home page.− Without login, the normal user cannot see more than simple description on the project and the copy rights and the year of modification.− When the user fills up the login form, and he/she is authorized to enter to this web.The web then moves the user to the indoor localizer page.− The indoor localization page contains number of persons listed, and followed by two buttons; the search and showing buttons.− The user should select one object at a time to search for its place.In this paper, only one object is available but for the future work, this project can cover large number of them.− After choosing the person's name, the user should click on search location button, which contains the hardware controlling and model prediction results.− The other button, location button, displays the captured images that had been interred to the model for testing reasons.

Hardware integration level
This section clarifies how the equipment parts are gathered, associated and utilized together as a total working framework for indoor item restriction robot.It contains two levels, the initial segment is connected for structure the vehicle segments, while the subsequent level is in charge of the controlling system for the vehicle and the rest segments.

Car parts gathering and arrangement
The principle segments utilized for setting up the mechanical vehicle are appeared in Figure 6.These parts are Raspberry Pi III model B, the L298N motor driver, the ultrasonic sensor, the voltage steps down regulator, the Li-Po battery, and the RPi camera [20][21][22][23][24][25]. Figure 6.The hardware requirement for car module

Robotic car movement and obstacles detection
Two ultrasonic sensors, fixed at the front and back of the car, are used to detect the obstacles in the car's passage way.The detection is accomplished by measuring the distance between the car and the faced obstacles.It is important to note that thel software algorithms are designed based on the proposed software engineering model.The proposed algorithm of car movement and obstacles detection is shown as a flowchart in Figure 7 and the next lines describe it in details: − For forward movement, the RPi fully stops the car, if the measured distance is smaller than or equal to (100 cm).The car is kept waiting for 5 seconds (secs).− After 5 secs, again it checks if the measured distance is increased.This means that the obstacles are moved, then the Raspberry Pi orders the car to drive again, until the measured distance is smaller than or equal to (100 cm).− In case of the 5 seconds are passed and the distance still the same, this means that this obstacle may be a wall or any stable object.− Now, the car is stopped, and it starts the procedure of capturing and classifying the images.− If the measured distance is smaller than (80 cm), this means that the robotic car is too close to the wall, is to take an image in a clear way the car must drive back at least for 20 cm, which can be equipped by moving the car backward for (1 sec).− For backward movement, The Raspberry Pi fully stops the car, if the desired time for backward movement is passed, or if the measured distance is below than or equal to (30 cm).

Classifying images procedure
After the robotic car has been reached to the wall successfully, the Raspberry Pi starts in capturing images.This procedure is shown in Figure 8 as a flowchart that explains the proposed algorithm.The image classification algorithm is proposed based on the software engineering model.The following lines are presented to describe this algorithm in details: − The first step in this procedure is to load the previously saved model in the RPi as a python script, by a special PHP formula in the website.After the user clicked on the search location button, and the robotic car found a wall (or as it thinks, it is a wall).− The RPi orders the camera to take the first image and save it in gallery file inside it.− Then it turns the car to the left with a 15 degree for changing the image angle, and again the camera takes another image and save it with the first one.− Again, the RPi turns the car back to its original location by turning it to the right with the same degree (15).− Then it moves the car to the right with the same mentioned degree, for taking the last image in another different corner.− At this point the system has three different images for distinct corners.These images enter to the loaded model as real time test images.The obtained three accuracies form three images are compared with each other.The maximum accuracy of these accuracies is chosen as the actual result of the classification.− In the proposed algorithm, a desired accuracy is considered to be 70%.− If the resulting image classification accuracy is equal or higher than this desired accuracy, then the classification is accepted and the image is an actual wall.Then this image should be assigned to the corresponding room and takes it as the final result.Otherwise, the RPi ordered the car − to turn to another wall.This is done by moving it to the left with 90 degrees.Then the procedure is started from the beginning, starting from robotic car movement and obstacles detection.

THE EXPERMENTAL RESULTS
To determine the efficiency of the proposed system, many experiments are introduced on different case studies.As previously mentioned, the system contains a website, in which, the user interfaces with the proposed algorithms.In this section, different case studies of different situations and different locations are discussed in details.The first step in all these case studies is to login to the system through the login page, for authority confirming, so this step is introduced first followed by the case studies.

Authorization checking
The user should be authorized to be able to enter to the page of localization in the website.Otherwise, the user cannot do anything with website except seeing the information that are shown in the home page.Therefore, and as a beginning in the localization procedure, any experiment or case study in the proposed system passes through the following lines: − In the starting of the website, a login button is appeared.The user should click on it to view the login form.
It has been noticed that there is no sign in button.For security and privacy purposes, no one can sign in to the system without authorization.

Case study five
This case takes the probability of putting new objects, which were not trained in the trained dataset.This experiment shows that the designed system can classify the walls to its corresponding room even if new objects are added like new chair, table, or new curtains.The same procedure of section B is executed here, with the addition of some obstacles as mentioned in section D.Moreover, a new curtain is added to the wall as shown in Figure 13 (a).This experiment is introduced in the admin room, and the classification results have shown that the probability of the images to be in the admin room was 82%.This proves that the proposed system has a great ability to classify any building as shown in Figure 13 (b).
The results of implementing the proposed system for different room classifications are listed in Table 4.It has been shown that many predictions are right.Many experiments had been introduced for this project, covering the eight walls, each experiment gave a different probability result.The results have shown that from every ten tested locations only one location is false.This means that the project false predictions are less than or equal to 10% which is acceptable loss percentage for a system starts with dataset from scratch.

CONCLUSION
An affordable indoor localization system based on deep learning was introduced and implemented to classify locations inside a building.The proposed system consisted of two parts: the classification algorithms and website building part (software part as well as the mini-robotic car (object) part (hardware part).The proposed algorithms were designed in deep learning technology with Keras library and CNN.A training model was created and trained, then the model was saved inside the Raspberry Pi SD card as a python script.The website was designed with HTML and CSS with the help of PHP for inserting the python scripts easily.Different experiments were introduced in order to test the proposed system.The results of these experiments have shown that it is easier to control the car by using only the Raspberry Pi control signals with high performance in term of accuracy and precision.

Figure 1 .
Figure 1.Block diagram of the designed software engineering model [2].

Figure 4 .
Figure 4. CNN architecture of the proposed model

TELKOMNIKA Telecommun Comput El Control −
Software engineering model based smart indoor localization system… (Muayad Sadik Croock)1911 Each image is passed through the loaded model and reach to a certain classification accuracy.

Figure 12 .
Figure 12.(a) Moving the car to the next wall, (b) The final classification results of the case study four

Figure 13 .
Figure 13.(a) Adding a new object, (b) The final classification results

Table 1 .
[14]view of the collected dataset The next step is to prepare these datasets to be ready for use in the training model.The data augmentation is used as a tool that is available in any deep learning toolbox.It is done with Keras library  ISSN: 1693-6930 TELKOMNIKA Telecommun Comput El Control, Vol. 18, No. 4, August 2020: 1904 -1916 1908 using a class called image data generator[14].This class applies a random invariance, such as images cropping, scaling, shearing, and flipping as shown in Table2.

Table 2 .
Overview of the used data augmentation techniques

Table 3 .
Model training results corresponding to each epoch  Software engineering model based smart indoor localization system… (Muayad Sadik Croock) 1909