A MODEL VISION OF SORTING SYSTEM APPLICATION USING ROBOTIC MANIPULATOR

Image processing in today’s world grabs massive attentions as it leads to possibilities of broaden application in many fields of high technology. The real challenge is how to improve existing sorting system in the Moduler Processing System (MPS) laboratirium which consists of four integrated stations of distribution, testing, processing and handling with a new image processing feature. Existing sorting method uses a set of inductive, capacitive and optical sensors do differentiate object color. This paper presents a mechatronics color sorting system solution with the application of image processing. Supported by OpenCV, image processing procedure senses the circular objects in an image captured in realtime by a webcam and then extracts color and position information out of it. This information is passed as a sequence of sorting commands to the manipulator (Mitsubishi Movemaster RV-M1) that does pick-and-place mechanism. Extensive testing proves that this color based object sorting system works 100% accurate under ideal condition in term of adequate illumination, circular objects’ shape and color. The circular objects tested for sorting are silver, red and black. For non-ideal condition, such as unspecified color the accuracy reduces to 80%.


INTRODUCTION
In the world of science and technology, more seems never be enough.For the sake of intelligent improvement to impersonate human's capability, not only motoric skill but also cognitive abilities, such as visual perception, the system to accomplish perception tasks tend to be more complex and technology rich.
The real challenge faced in the lab as well as in industrial setting is how to improve existing sorting system in the Moduler Processing System (MPS) [1], [2].The MPS typically consists of 4 integrated stations called distribution, testing, processing and handling.Old sorting method uses a set of inductive, capacitive and optical sensors do differentiate object color in the testing station.Handling is done by using a programmed manipulator, as available in the MPS manual book.No vision capability exists in the system to improve its performance and flexibility.Replacing or complementing these 3 sensors with visual sensor is like enabling the eyes to a blind person walking with stick.This is the main objective and contribution of the paper.
Computer vision is developed and generally used for such application that requires fundamental visual functions, capturing and recognizing visible objects or pattern just like in sorting system.In object sorting system equipped with computer vision, the image of environment is captured and processed to obtain information as dictated by system requirements.It usually takes data about the characteristic of workpieces, location and even color.This paper is intended to describes a color based sorting system by employing the application of image processing to recognize objects' shape and color and then to group the object colorwise.Similar but more basic grading system of lemon using vision is in progress as reported by Khojastehnazhand [3].

PROPOSED METHOD AND ALGORITHM
Image can be assumed as the visualization of what vision senses that is captured by image capturing devices.Image is considered as a two dimensional function with variables that represent the spatial coordinate.It holds information about color as well as shapes.In color image, RGB color model mixes those three prime color components, red, green and blue, to produce another color [4], [5].Image capturing and processing have been used widely in diverse applications, such in medical and surveillance applications [7], [8].

Circular Shaped Object Detection Method using Hough Transform
For object detection, Hough transform is a method widely used for object recognition applications [8]- [10].Roughly, it checks the likeliness between the cluster and the object model to find what cluster corresponds to that object in an image.At the beginning, the original one, Hough transform concentrates only on finding line but it has been developed to find other simple shapes, such as circle and ellipse.In a plane, circle is characterized with center coordinate (a,b) and radius r, as illustrated in Figure 1.

Figure 1. Circle representation
The method to find the circle shape starts with preprocessing.The input image has to be processed to be prepared, using smoothing.Then the input is transformed into circle parameter with radius and center relation as in Eq.1.Classical way, the circle is characterized by using three parameter (x, y, r).The circular shape can be defined after edge detection phase.Edges detected will contribute circles with certain radius.The point with the most interception from circles is the center coordinate.

Object Handling Method using Robot Manipulator Mitsubishi Movemaster RV-M1
A package of Mitsubishi Movemaster RV-M1 consists of a robot arm with gripper hand, teaching box, drive units and also the manual [1], [2].It is an industrial articulated 5-axis robot with waist, shoulder, elbow, wrist pitch and wrist roll, as part of Modular Processing Station (MPS) in the laboratory, as shown in Figure 2. It can be controlled by using a personal computer configuration where all commands and decisions are made in computer as direct execution mode.RS-232C interface is used between the PC and the drive unit for this purpose, as shown in Figure 3.

RESEARCH METHOD
Figure 3 shows two main parts of the subsystem, image processing and handling subsystem.In image processing, the webcam captures in realtime the condition of workpieces.This picture captured is passed through noise reduction phase before it is processed to get data about position and color.The data from this process will be transmitted as part of commands to the drive unit to instruct the Mitsubishi Movemaster RV-M1 to move in certain pattern based on the color and position of the workpieces under the sorting process.That sequence of commands will pick the workpieces, one by one, from the pallet and stack them in the magazine based on their color.The overall system flowchart is given in Figure 4 (a), including its sorting process flowchart subprocess in in Figure 4

Image Processing for Shape and Color Recognition
The task of image processing part is to get information about position and color of the workpieces.The position is determined from the center coordinate of circular object and color information.Before the circle detection phase, the input image, the workpieces on the pallet, has to go through noise reduction process as it is illustrated in the Figure 5.
Noise reduction method cleans the input image in order to get rid of noise.In this process, the input image is segmented to get only the objects to be focused on.The pallet has nine circle holes for the workpieces spots as shown in Figure 6.Handling table has its own surface contour.Background subtraction is a method to compare the background image and the real current image to find any different.Two images are taken; one image grabs the condition of empty pallet and with the assumption that its initial condition does not change.Another image is taken for the condition after the workpieces come in the pallet randomly.Image differencing technique subtracts every pixel in the first image with the corresponding pixel in second image then remappes the result in a new image.It contains workpieces information changes due to the existence of workpieces in the pallet.
In some cases, additional clean up method has to be applied to assist the background subtraction process.Erosion and dilation are common techniques to take out small noise.Erosion turns off all pixels that were on and dilation works the other way round.The imperfect subtraction may leave small noise around the workpieces that should not be there.Next, the clean difference image is coverted into a gray image and get smoothed by Gaussian kernel.Smoothing process reduces rough edges and the image is ready for circle detection.
Hough circle transform method is used.In OpenCV library, cvHoughCircles is the function to identify circular shape [11].Circle parameter is center coordinate and radius.This circular shape detection requires more memory and time for the algorithm to be run compared to the line detection.Hough circle transform employs gradient method to find the circle.It finds the local gradient and checks the possibility of circle shapes.Hough transform begins with an edge detection process.Completion of circular shape detection is followed by color recognition.Color recognition task is a process to determine the dominant color in the circle as it is illustrated in Figure 6.The method to define the color in this paper is by comparing the 3 basic color values of R, G, and B one to another.Figure 6.Color analysis The workpieces have three different actual colors, red, silver and black that has to be recognized.For each color, the combination of RGB value has to be analyzes to determine the color of detected workpieces.The value of the RGB depends on the intensity if the image.Percentage method might be friendlier for this job.The idea of this method is by taking the value of each RGB in a pixel and then calculates the percentage of each color in that pixel.This calculation results in how much portion of that basic color out of the whole intensity in one pixel.
(2) This percentage representation gives better illustration about the amount of basic color in one pixel compare to the real value of the color.The percentage value is not taken not only in one pint but in area around center coordinate.The averaged value will be used as final color result.

Handling and Sorting
As shown in Figure 7, the handling subsystem does the pick-and-place mechanism and it is repeated untill no workpieces is left.Except for continous cycle, it only takes one single object.The input for the Mitsubishi Movemaster RV-M1 is a series of commands that creates movements from one position to another position.The position comes from the workpieces location and color.In Figure 8, the center coordinate from cvHoughCircles is classified into nine possible positions in the pallet.The pallet is divided into nine areas for numbering the position.Marks are defined in the pallet to classify to what position that the center point belong to.For example, when the pixel of center point is smaller than x 1 and y 1 then the position number is 11.
The information about color are interpreted in series of position too that will define to which magazine the workpieces will be stacked.The numbered magazines and workpieces are illustrated in Figure 9.Each magazine holds one color for three different specified colors.The fourth magazine is dedicated for rejected object when the color's characteristic is neither red, silver nor black workpieces.Those two numbers, pick positions in pallet and place positions in magazines, are used in the command combined with another command to have a complete pick-and-place mechanism to move randomly placed workpieces from source pallet to sorted destination magazine.

RESULTS AND DISCUSSION
All processes and experiments are applied to the specified objects recognition which have circular shape in the surface and three different color which is red, black and silver.The overall system is shown Figure 3

Single Mode Operation Using Image Differencing
In a single mode system, a background image of empty pallet is taken only one time just right before the current image.Image substraction is applied to the current against background image.The manipulator handles all workpieces detected in the image.Figure 11 shows pallets with full and partially full workpieces to be compared with an empty background pallet.Table 1 shows the results of detection.After experiments of various fully and not fully pallet, the results indicate that this image subtraction and object detection gives 100% accuracy.None of the circle was found in empty pallet nor outside the pallet.

Continuous Mode Operation Using Image Differencing and Erosion
In continuous mode selection, the sorting system handles one object and moves back to initial position to capture another image.The background is taken one time in the beginning of the whole process.The current images are taken after the manipulator has finished handling one workpiece.As the manipulator has done movements, the exact webcam position could shift slightly due to mechanical inperfection of the robot manipulator.In this case, there will be image shifting effects that act as noises and disturb the circle detection routine.The image shifting gave noises as the background subtraction is not clean enough in removing all noises.The solution was erosion technique to clean the image [4], [5], [11].
The Result can be seen in Figure 12.As shown Figure 12.a, before the erosion the image substraction is rather noisy and circle detected was not accurate.Noises outside the pallet were also signified as a circle.After the erosion was applied to the image, Figure 12.b, the detection process performed accurately.All objects were detected, as there was no more noise outside the pallet area.Erosion function corrected successfully the circle recognized along with the noise removal.Several experiments were taken to test the reduction as a result of erosion method.The robot manipulator was moved to some pick-positions and returned to homeposition.Table 2 contains the data of circle found before and after erosion for continuous mode.The erosion reduced all previous circles detected when the pallet was empty, no workpieces.For the case where there were workpieces, it resulted in fully accurate detection.However, there was one condition that detects no circle while actually there was one object.This error was caused by the illumination factor.The position of illumination source caused the upper right position to be darker since it was a little bit covered with shadow of robot hand.That dark area becomes even more surrounded by zeros pixel since the erosion broaden black spot.By setting the table to face the illumination source, the shadow of robot hand will not affect the pallet location, thus resulting in better detection.

Color Analysis Using RGB Procentage
Several tests are used to check the color detection capability based on RGB percentage formulae derived in Eq. 2. The results of nine times RGB analysis in area around center coordinates give consistent color detection for 3 specifiend colors, as in Table 3 and Figure 13.The result in Figure 13 relates color detection to its RGB values as follow Red objects : R > B > G Black objects: This finding is further tested to the pallet with combined color of workpieces.Figure 14 is the input image and Table 4 is the record.

Command Transmission and Workpiece Handling
The robot was setup as in Figure 3 and pictured in Figure 10.It was ordered to move workpieces based on the sequence of some robot commands from the laptop, as listed in Table 5 [2].It gave wrong result when all commands were sent one after another without delay.So, delay between commands has to be set to allow robot to finish one movement before it can accept further commands.Short movement, such as GC for gripper close, took less time to execute.On the contrary other commands, such as NT for nesting, took much longer time to execute.Variable delay using sleep() command was needed for each command, but it reduced the programming flexibility.A more flexible approach was robot position checking command using WH (where) to solve the delay issue.The robot was given one command to move to one position, and then WH function was sent.It replied back information about the coordinate of its current position.The laptop continuously checked this coordinate until WH returned the coordinate of the destination position.This destination position indicated that the robot was ready to accept another further command.This WH position enquiry commend gave better result in term of efficiency.It reduced wasted time for both long and short duration command.

CONCLUSION
The new sorting system to separates circular objects based on their color using OpenCV library has been successfully accomplished.The image processing result was converted chiefly to the commands that drive the handling subsystem, Mitsubishi Movemaster RV-M1.There are two main steps in image processing part, objects detection and color recognition.Hough circle transform was capable to detect circular objects.Enough illumination and minimum noise condition has to be well configured to obtain accurate results.The information about shape, position and color, was processed into sequence of commands that were transmitted to the drive unit of Mitsubishi Movemaster RV-M1 as handling device.For every command, Mitsubishi Movemaster RV-M1 gave back current position of the robot for checking purpose to determine whether the robot was still executing or was ready for next command to avoid command overlapping.
In conclusion, the system has successfully performed handling station task, namely pick-and-place mechanism with the help of vision application.Two modes were provided, single and continuous as a selection of execution.As tested, testing with non specified color gave 80% accuracy, while testing with specified color resulted in 100% accuracy.Under the ideal condition, the system can detect objects and operate with 100% success rate.In the end, it improves the existing sorting system with new vision capability, complementing or even replacing the old method.

Figure 7 .
Figure 7. Handling Flowchart Figure 8. Position Designation for Pallet and Figure 10.There are two main modes of experiments, namely single and continuous modes, as illustrated in the flowchart of Figure 4, 5 and 7 earlier.

Figure 13 .Figure 14 .
Figure 13.Charts of color analysis in area around center coordinate: (a) red, (b) black, (c) silver

Table 1 .
Results of single mode circle detection

Table 2 .
Results of continuous mode circle detection

Table 3 .
Color analysis in area around center coordinate: (a) red; (b) black; (c) silver

Table 4 .
Result of final color analysis of Figure14