Intelligent Avatar on E-Learning using Facial Expression and Haptic

The process of introducing emotion can be improved through three-dimensional (3D) tutoring system. The problem that still not solved is how to provide realistic tutor (avatar) in virtual environment. This paper propose an approach to teach children on understanding emotion sensation through facial expression and sense of touch (haptic).The algorithm is created by calculating constant factor ( f ) based on maximum value of RGB and magnitude force then magnitude force range will be associated into particular colour. The Integration process will be started from rendering the facial expression then followed by adjusting the vibration power to emotion value. The result that achieved on experiment, it show around 71% students agree with the classification of magnitude force into emotion representation. Respondents commented that high magnitude force create similar sensation when respondents feel anger, while low magnitude force is more relaxing to respondents. Respondents also said that haptic and facial expression is very interactive and realistic.


Introduction
The three-dimensional (3D) tutoring system is one of an innovative way to change the conventional teaching methods.However the realistic 3D tutoring system requires a human element like emotion expression to give impression to the users.Therefore, studies of artificial psychology have been conducted to create an interface that is able to satisfy human during their interaction with computer.Human representations like virtual assistants or virtual humans in the virtual environment are considered as an interface that can make harmonious relationships between the human and the computer [1].That is why a lot of existing applications such as Second Life or games like the SIMS manipulate avatars in their systems.Based on the research findings, the previous studies are lacking of presence in terms of virtual humans.It can be caused by several factors such as: the complexity of virtual human model itself i.e. emotional expression [1], or incoherent animation reaction [2].Usually, emotion is expressed by using facial expression, voice intonation or even using hand gesture.The facial expression of emotion was initiated by [3] and today, facial expression is still a challenging area to be explored.[4] have created a standard guideline for emotional facial expression of human named facial action coding systems (FACS).Inside FACS, there are action units (AUs) which is representing particular area of human face.Facial expression also triggers facial animation improvement such as paramaterized facial model [5], facial rigging based on blend shape [6], facial rigging using cluster [7] and facial rigging users interface [8].
Haptics or 'sense of touch' attracts attention to be explored -especially Haptics that is related to emotional expression [9], [10].Our work will utilize haptic tactile vibration as an emotional expression of virtual human or avatar to give the users more sensation during their interactions or even to stimulate the users to become familiar with virtual environment and its contents.Through vibration mapping technique, human emotion colour scheme will be mapped into magnitude vibration force to stimulate users to understand particular emotional expression of avatar such as anger and happiness.If we look further inside 3D tutoring systems, there are some parts such as avatar, environment etc. that contribute to create realistic effect to the virtual reality game itself.Avatar becomes a crucial part of virtual reality game.Researchers try to make virtual human (avatar) that is able to show a particular emotion using facial expression, sound effect and even with sense of touch [11][12][13].[12] create virtual human that can express their emotion using body touch, even supported by the alteration of environment like hue , brightness and saturation.[13] concentrates on how virtual human can behave like human being by increasing their ability to make a more interactive dialogue.
Previous researchers like [14] describes that non verbal communication in collaborative virtual environments (CVEs) can be communicated using face, gaze, gesture or even body posture.Now, researchers are doing some expansion in terms of providing human representation to increase the interaction and communication between computer and human.
[1] also mentions that there are two main problems with creating a virtual human i.e. construction of emotion and generation of affection model to be implemented in the virtual human (to improve their presentation).The avatar does not only represent human in terms of physical representation, it also needs some believability context.According to [15] the current avatars need to be improved due to their lack of believability.Rojas [2] propose individualization method by putting personality, emotion and gender into the avatar.Next, the avatar is turned into a character and able to express their emotions by involving the act of touching among two characters [12].Other researchers like [16] make some innovations in expressing the emotion of virtual characters by putting aside body parts.They use elements like light, shadow, composition and filter as tools for conveying the characters' emotion.In terms of human emotion research area, the basic human emotions have been recognised and formatted into standard forms.For example, an emotional facial expression can be detected from the eyes and lips of a human face [3].Other researchers try to recognise emotions from several ways such as, colour [17][18][19][20], haptic device [9], acoustic [21], music [22] and dance movement [23].
According to our preliminary data collection into senior high school at Malaysia, most of them still not familiar with the 3D tutoring system and when they have been introduced into some existing 3D tutoring systems they still not so excited.Based on this fact, we have proposed a new and innovative solution by integrating three of human senses into multimodal expression (visual and haptic) which is called as Emo-Avtor.The proposed system also facilitate user with different interaction to help them during communication process with Emo-Avtor.The motivation of research on this paper is also inspired by haptic ability that can give different sensation compared to visual stimulation and haptic stimulation.As we mentioned before, most users are much not affected by facial expression of avatar because they can only see the changes of avatar facial expression and various tone of voice intonation.These situations motivate us to make a bridge between avatar and user (human) when they communicating each other.

Research Method
Figure 1 is our system demonstration to Senior high school at Malaysia.Most of them is very excited when they interacting with our system.Emo-Avtor is an integration of two human TELKOMNIKA ISSN: 1693-6930 Intelligent Avatar on E-Learning using Facial Expression and …. (Ahmad Hoirul Basori) 117 senses: visual and haptic.Haptic is requiring preprocessing to classify the magnitude force into suitable range for emotion expression.The classification is based on colour theory which is mapping the human emotion characteristic into RGB mode, and then RGB property will be converted into magnitude force of vibration.Why the mappings need to be passed through RGB not directly from emotion into magnitude force?This is due to the previous research on magnitude force only touch the human response not specific to the emotion types.That's why we do mapping process through colour which is contain emotion value and matching the colour value of magnitude force to check whether the mapping process is right or not.
The input of Emo-Avtor is obtained from two main sources: First is mind controller that will read the user brain activity during the interaction with Emo-Avtor and it will change the avatar emotion expression in real time.This mean when user express anger then avatar will do the same expression (anger).The second input is come from 5DT glove which is able to read the movement of finger.There are two shape of hand gesture that able to be recognized in this system.One is creating fist shape (anger) and the other one is show up the thumb (happiness).On the other hand, the output of Emo-Avtor is displayed through the avatar which is able to express their emotion through visual and haptic.Furthermore, we also contribute on designing and testing wearable haptic to give user incredibly realistic emotion sensation.Figure 2 describing the overall architecture of Emo-Avtor system.Pre-processing phase is only used once at the first time to generate the appropriate magnitude force for each emotions and it also evaluated according to the emotion condition.This data will be used for the facial expression real time simulation.

Calculation of Constant factor (f) and Mapping Magnitude Force
From previous study vibration frequency can be classified into certain ranges of magnitude force.To plot the given vibration frequency into emotion, it needs the following equation where the constant factor (f) denotes colour and vibration.When the intensity of red colour needs to be changed, this means that RGB value of red is hold in 255 and hue=0 while saturation is moved from 100% to 0% whereas RGB value green and blue is moved from 0 to 255.The mathematical formula below is produced by investigating the colour and vibration behaviour. note:

Calculation of Constant factor (f) and Mapping Magnitude Force
The conversion process is illustrated in Figure 3, where vibration magnitude power increases or decreases linearly according to colour intensity.The adjustment of colour intensity will affect the vibration power.2), the classification of emotion according to magnitude force is listed in Table 1.In Table 1, the emotion can be mapped into magnitude force value by transferring the emotion in colour perception into magnitude force.High magnitude force more than 5000 is potential for carrying the 'anger' emotion and magnitude force lower than 5000 is similar to 'happiness' emotion characteristics.The result also matches with the previous research [17][18][19][20] which described thet red colour is able to represent negative e.g.anger and positive emotion e.g happiness.By applying the formula, we successfully create the conversion procedure from colour value into vibration power.We found that the speed of

The validation of magnitude force effect to human body
The used joystick in our experiment has two motor rumbles which are able to produce certain magnitude force and frequency.The haptic in this study is called wearable haptic which is placed in the jacket and it's controlled wirelessly as shown in Figure 4.The subsection will prove empirically the effect of magnitude vibration power of joystick to the human as well as to prove the relation between magnitude force and human emotion.Joystickhas magnitude forces i.e. the force able to produce certain rotational speed or velocity (V) and acceleration (A).There are several physics formulas that are very important to this experiment.According to investigation on joystick for the thesis, the joystick used is xbox windows controller and Teysun Joystick.All regular joysticks have the same size of motor rumbles.

( . . ) 2
A r m sMagnitude r m sA = The experiment is conducted by calculating the number of rotation of motor rumbles in some periods of time.Big motor rumble has the characteristics of producing hard vibration and strong sensation, while small motor rumble tends to generate smooth sensation.The two motor rumbles can be rotated simultaneously to generate strong vibration.The number of rotation will reveal velocity, frequency and acceleration of the joystick.Based on the r.m.s acceleration, the vibration will expose the unpleasant and pleasant feeling to human.The outcome will provide empirical evidence to strengthen the mapping technique.Table 2 lists the acceleration of big and small motor rumbles and the total A and r.m.s.A is considered as a parameter for evaluating the effect of vibration to human and its relationship with emotion characteristics.According to the standard deviation of mean A from big motor and small motor, the longest dispersion of mean A is 0.29 and 0.20 for r.m.s A. Figure 5 is an illustration of Acceleration behaviour of both motor rumbles of joystick, total of mean A and total r.m.sA and standard deviation between both of them.The graph of total mean A is different from the others as it rises rapidly since the starting point because the total mean A is the sum of mean A big motor and small motor.The other graph that is closely similar is the total r.m.sA with mean A where r.m.sA reaches a peak with 1.28 ms-2 r.m.s and it decreases until zero.This result needs to be compared with the previous result to prove the similarity behaviour of vibration effect and emotion classification.There are numerous previous researchers who have described the implication of vibration to human.Oborne and Clarke (1974) in [24] have classified the vibration based on mean magnitude and semantic scales.
• Mean magnitude (ms  2, it has been described that the total r.m.sA is able to reach the peak until 1.28 ms  This uncomfortable feeling is similar with anger emotion behaviour which makes human feel unpleasant i.e. annoy to them.While the Magnitude Force of 5000 and below are prospective to illustrate the behaviour of happiness emotion which causes very comfortable or pleasant feeling to human.With the results proposed in this chapter, the goal of producing emotion-vibration mapping technique succeeds and proves that the proposed technique is able to map the emotion into magnitude force of vibration and it also proves that the classified technique is suitable to emotion characteristics such as anger and happiness.This result is also supported by Bailenson et al [9] that anger has more speed i.e. faster than happiness.

Results and Discussion
This study has recruited: 8 male and 13 female that comes from undergraduate, post graduate student (Master and PhD).Each student is required to make interactive interaction with system and after interaction student will be asked to answer some questions in questionnaire.Question will base on data profile, emotion background, experience emotion, interaction, and benchmarking.Figure 6 show that 15 users strongly agree that emotion expression can be represented by magnitude force of vibration.The surprising result comes from users' rating that high vibration is equivalent to anger emotion; this outcome is similar to our approach result on Table 2 that anger can be represented with high magnitude force.The other user interaction with our proposed system is shown in Figure 7.The users interact with avatar to change the facial expression of the avatar, then avatar will give feedback to user using their facial expression, voice and magnitude force of vibration through joystick.This glove is designed for controlling the intensity of facial expression of 3D avatar by moving the finger into fastidious position e.g. if user is trying to make a "fist" by glove, it will be interpreted as "anger" emotion then the avatar will change simultaneously.The avatar emotion is shown by facial expression that control through Action Units around facial area.In term of reading the signal from ind controller, there are several paramaeter that used in this study.Those parameters are Alpha (1, 2, 3) and Beta (1, 2, 3) as the indicator of brain activity.The signal of Alpha and Beta is divided into four main zones i.e.Z1, Z2, Z3, and Z4.The brain signal will be divided into these four zones.Z1 and Z2 are associated with happy emotions while Z3 and Z4 is associated with anger; the difference between both of them is the strength of the emotion.If brain activity remains strong until the signal enter Z2 zone, it is considered as a very strong happiness.Furthermore, 5DT glove will also calculate the intensity of each emotion as shown in Figure 8.

Conclusion
The aim of this research is to implement haptic as me-dium to streghthen the emotional expression of avatar for 3D tutor in E-Learning system.Our conducted ex-periments proved that user manages to have strong feeling and impression while they interact and communicate with the system.The study has proved that high magnitude force of haptic device like joystick capable to create emotional sensation which is classified by intensity of RGB colour.Furthermore, it is able to be syn-chronized with facial expression of avatar as well.From user study, the feedback from user is very exciting while 67% user give strong and positive response to the system.In addition, 15 users from 21 participant (71%) agree with classification of magnitude force into emotion representation, they said high magnitude force create similar sensation when they feel anger.While low magnitude force is more relaxing to them.This system is believed to bring different ways of teaching emotion to the students.Most of them said that they got better understanding during their interaction with proposed intelligent avatar.

Figure 1 .Figure 2 .
Figure 1.Demo and testing to senior high school students for constant Vmax = maximum value for vibration (10000) RGBmax = maximum RGB value for Red, Green or Blue Colour (255) f = 1000/255 = 39.21

Figure 3 .
Figure 3. Flow for mapping color to vibration

Intelligent
Avatar on E-Learning using Facial Expression and …. (Ahmad Hoirul Basori) 119 motor rum-ble increases linearly with vibration frequency as well.For example, anger emotion will trigger high speed which proves the initial experiment conducted by Bailenson[9].
T= duration or period X=displacement, the distance of body or hand from rotational source r= radius of motor rumble ω=angular velocity V in SI unit is symbolized with metre per second (ms -1 ) A in SI unit is symbolized with metre per second (ms -2 ) Mean magnitude or r.m.s Magnitude in symbolized with root mean square metre per second (ms -2 r.m.s) ISSN: 1693-6930 TELKOMNIKA Vol. 9, No. 1, April 2011 : 115 -124 120

- 2 r
.m.s.When it is compared to the semantic result from Oborne and clarck (1974), the result shows a strong relationship with emotion classification with the TELKOMNIKA ISSN: 1693-6930 Intelligent Avatar on E-Learning using Facial Expression and …. (Ahmad Hoirul Basori) 121 magnitude force above 5000 is potential to carry fairly uncomfortable and uncomfortable feeling to human.

Figure 5 .
Figure 5. Acceleration graph for big and small motor rumbles.

Figure 6 .
Figure 6.User rating ranking emotion representation strength

Figure 7 .
Figure 7. User Interaction with the avatar using haptic device.

Figure. 8 .
Figure. 8. Zone classification of Alpha and Beta signal 123

Nia Mind Controller 5DT Glove Facial Expression Generator
Calculate constant factor (f), from colour RGB to magnitude force Mapping magnitude force according to constant factor Compare colour from mapping result to emotion characteristic Create classification table for magnitude force according to emotion behaviour Input

Table 1 .
Conversion from emotion into vibration power based on colour theory note: R: red, G: green, B: blue H: hue, S: saturation, V: value

Table 2
Acceleration of big and small motor rumble