Overview of Social Cognitive Ability from the Perspective of Facial Expressions and Context — Estimating Emotion Based on Context Essays and Facial Expressions —

Career Design Laboratory for Gender Equality, Kanazawa University, Kakuma-machi, Kanazawa 920-1192, Japan 1The National Institute of Vocational Rehabilitation, 3-1-3 Wakaba, Mihama-ku, Chiba 261-0014, Japan 2Graduate School of Engineering, University of Fukui, 3-9-1 Bunkyo, Fukui 910-8507, Japan 3Faculty of Regional Science, Tottori University, 4-101 Koyama-cho Minami, Tottori 680-8550, Japan 4Faculty of Education and Regional Studies, University of Fukui, 3-9-1 Bunkyo, Fukui 910-8507, Japan

In recent years, increasing numbers of children have unbalanced emotional developments and sociability, particularly individuals with autistic spectrum disorders (ASD), and have difficulty inferring other people's feelings from their facial expressions and contexts, and therefore, they are less able to respond with prosocial behavior.Thus, teaching the connection between facial-based or situation-based emotions and prosocial behavioral responses is needed.In this research, we provided essays describing the context and facial expressions of a given character, and gave participants tasks to estimate the characters' emotion under two conditions: emotions described in the context essay and given facial expression were congruent, or they were incongruent.Furthermore, we tracked frontal lobe activity that was deeply associated with high cognitive functions using near-infrared spectroscopy (NIRS) and examined whether the frontal lobe activity related to emotion estimation differed as a function of the sympathy level.

Introduction
In recent years, the importance of social cognitive ability in the workplace, particularly the need for communication has gained attention.However, increasing numbers of children have unbalanced emotional development, problems with personal relationships, sociability, and behavior, and have difficulty adapting to classrooms.To solve these problems, it is imperative to plan educational programs specifically designed by considering each child's ability and developmental condition, such that children can adapt to their daily environments and overcome their problems.However, it is difficult to train social abilities and problem solving abilities, since social cognitive abilities are context dependent and related to emotions.Therefore, the current support system for children with social problems needs to provide coping strategies after children experience problems in social situations.Thus, the development of a human interface that would allow children to develop social intelligence, undergo social experiences, and learn sociability through human-machine interactions is required.For example, by applying "learn by teaching," interactions, child users could teach social interaction to a machine while the machine plays a learning role in a given situation.The machine would also need to comprehend individual characteristics of each child's basic social cognitive abilities.Such a machine requires flexibility to provide feedback to promote the development of children's sociability, based on their developmental level and reflecting their social intelligence, social cognitive abilities, and characteristics.The construction of smooth personal relationships with others demands the ability to estimate other people's psychological states, interpret them, and predict their behavior.Thus, it is imperative to clarify how normally developing children estimate others' emotions based on facial expressions and contexts.
Face recognition is a key concept in the autistic spectrum disorders (ASD) research field, so many previous studies on face recognition (1)(2)(3)(4)(5)(6)(7)(8)(9) have been conducted.In this study, we focused on sympathy as a psychological factor that is related to accurate emotion estimation and examined the difference in strategy for assuming emotions between children with high and low degrees of sympathy.Moreover, we investigated neural activities during emotion estimation.We specifically provided facial expressions and context essays that consisted of multiple cues for emotion estimation.Identifying incongruences between facial expressions and contexts, for example, a sad facial expression in a happy context, could provide a clue to a deep understanding of other people's emotions.In real life, there are many situations in which multiple cues, such as facial expressions and contexts, are used to estimate emotions.However, there have been few studies that have examined the processes involved in emotion estimation.
In this study, we provided essays describing the context and facial expressions of a given character, and gave participants tasks to estimate the characters' emotions under two conditions: (1) emotions described in the context essay and given facial expression were congruent, or (2) they were incongruent.Furthermore, we tracked frontal lobe activity that was deeply associated with high cognitive functions using near-infrared spectroscopy (NIRS) and examined whether the frontal lobe activity related to emotion estimation differed as a function of sympathy level.NIRS is a method for measuring frontal lobe activity, which is deeply related to high cognitive functions.

Participants
Participants were 14 normally developing Japanese children [N = 14, 8 boys and 6 girls, age range 11 year and one month to 17 years and 8 months (mean ± SD = 13 ± 2 years old)] in the study.The parents of the children signed informed consent forms in advance and accompanied their children.

Procedure of experiment
(1) The task was explained to the children.
(2) Children moved to a shield room and completed five trial tasks that require estimating the given characters' emotions from their facial expressions and the context.(3) Children's understanding of task details was verified and they wore the NIRS headgear.(4) Eight blocks with ten tasks in each block were given to each child.(5) The children took off the headgear and left the shield room, followed by the writing of the introspective report.(6) After completing the Sympathy Scale (question sheet), the experiment ended.

Stimuli and materials (1) Facial expression pictures
We took pictures of a 12-year-old boy and a 7-year-old girl showing pleasant feelings (happy) and unpleasant feelings (disgust) by using a digital camera.These four pictures of facial expressions were trimmed to an oval shape by eliminating the neck and hair. (

2) Context essays
The boy in the picture was given a fictional name, "Taro" and the girl "Hanako".We created essays describing the contexts in which these characters were experiencing emotions (context essays).We call each sentence that has a context as a 'context essay'.There were two types of emotional contexts, pleasant and unpleasant emotions, to evoke characters' emotional state.Ten sentences were created for each type of emotion (e.g., Taro was praised by a teacher and Hanako was scolded by a teacher) giving a total of 20 sentences.All the sentences started with either character, Taro or Hanako, as the subject.Twenty sentences for each character were created resulting in 40 sentences.
[80 tasks is "face" {(happy or disgust) × (Taro or Hanako)} × "context essay" (pleasant or unpleasant).](3) Stimulus presentation and measuring device A color liquid crystal display (LCD) monitor was used to present the stimuli.The distance between the participants and the monitor was approximately 140 cm.The participants responded using a numerical keyboard.(4) NIRS The change in oxygenated hemoglobin (Oxy-Hb) concentration in the blood during the task performance was measured using ETG-7100 (Hitachi Medical Corporation).By using probes that enabled measuring 22 ch in NIRS, near-infrared light (wavelength 700-900 nm) was irradiated to measure the change in Oxy-Hb concentration in the cerebral cortex.To measure the activity of the prefrontal area, a 3 × 5 probe holder was focused in the middle of the anterior, i.e., Fpz using the international 10-20 system, and then covered both sides of the prefrontal area (Fig. 1).The sampling frequency was 10 Hz, a 0.02 Hz low-pass filter was used to remove the effects of Mayer waves, and a 0.001 Hz high-pass filter was used to remove baseline drift.(1) The context essay was presented at the center of the computer screen (5 s).

Flow of experimental task
(2) Next, the facial expression picture was presented (mentioned in Step 1) in the middle of the screen (2 s).(3) A question asking to estimate the emotion of the character was presented on top of the screen, and below the question, a choice of three responses was presented (happy feeling, bad feeling, and others).The participants chose the best response from the three choices.At the bottom of the screen, another question was presented asking which cue the children used to estimate the emotion.Three choices (essays, facial expressions, and both facial expressions and essays) were given to choose the best answer.To answer these questions, the participants were required to accurately hold a numeric key corresponding to each choice for more than 3 s.(4) The processes from Steps 1 through 3 were repeated.One block consisted of 10 trials (approximately 6 min), and 8 blocks (approximately 50 min) of the tasks were conducted.The experimental condition was presented in a random order (Table 1).Two factors were examined in the present study, facial expressions and context essays: 2 types of emotion in the context essay (pleasant and unpleasant) × 2 types of emotion in facial expressions (pleasant and unpleasant), and four conditions.

Sympathy Scale
The Sympathy Scale was a questionnaire created by Sakurai to measure the level of sympathy.The questionnaire included 20 questions, and responses consisting of "yes" or "no" were recorded in a 5-point scale. (10)

Procedure
The experiment was conducted in the shield room.Prior to the experiment, the details of the experiment and the description of the NIRS device were explained to the children and their parents, and their consent was obtained.Then, the children were seated in a chair in the shield room and engaged in 5 practice trials using the numeric keyboard located on the right-hand side.After confirming that children completely understood the experimental procedure, they wore the NIRS headgear.To prevent interference to the NIRS signals by artifacts, the children were told not to move their heads during the experiment.
The children were shown a brief profile of the characters and instructed to carefully estimate the emotions of the characters, not the emotion of the children themselves, in the given contexts.The instructions given were as follows: "Please press the button of the color that corresponds to the choice showing the emotion of the characters by using the essays and facial expressions as a hint.Although facial expressions can be a big hint to read people's feelings, sometimes it is difficult to read their true feelings only from the facial expressions, such as bitter smiles and fake crying.Thus, please focus on both facial expressions and sentences, and read their mind".

Recording indices (1) Selection of each choice
For each experimental condition, the choice ratio regarding the clues used for the emotion estimation was calculated.In particular, the following equation was used to calculate the ratio.Choice ratio = number of trials that chose the corresponding choice/total number of corresponding trials (2) Response time for emotion estimation The time taken by children to choose the emotion estimation response (selective response time) was calculated as the time from the moment when the choice screen was presented to the moment when the participant pressed a key.
(3) Measurement signals of NIRS Based on previous studies, (11,12) oxy-Hb of measurement signals in NIRS varies less and indicates a higher repeatability than deoxy-Hb.Thus, in the present study, the analysis was limited to oxy-Hb.Using the analytical software, arithmetic means of the change in oxy-Hb concentration for each experimental condition were calculated.For the arithmetic means, the moment that a facial expression picture was presented was considered as the zero point, and 2 s prior to that moment was defined as the baseline.(4) Sympathy Scale For each question, in the order of high to low sympathy, a score was given as 5, 4, 3, 2, and 1 point.The sum of the points for the 20 questions was defined as the sympathy score.On the basis of the obtained sympathy score, children were divided into two groups: a high-sympathy group and a low-sympathy group.Children whose sympathy scores were higher than the median (61 out of 100 points) were grouped into the highsympathy group, and children whose scores were lower than the median were grouped into the low-sympathy group.The low-sympathy group included six children (four boys and two girls), and their mean ± SD was 56 ± 2.9 points.The high-sympathy group included six children (three boys and three girls), and their mean ± SD was 67 ± 5.7 points.Two children who had median scores were excluded from the analyses.

Data analysis (1) Selectivity of each choice
To examine whether the judgment of emotion estimation differed as a function of sympathy level, a two-way ANOVA, types of emotion (happy feeling, bad feeling, and others) × 2 levels of sympathy (low sympathy and high sympathy), was performed for selectivity about the choices of emotion estimation.
(2) Selective response time for emotion estimation To examine whether the response time for judging emotion estimation differed as a function of sympathy level, a two-way ANOVA, 3 types of judgment resource (essays, facial expressions, and both facial expression and essays) × 2 levels of sympathy (low sympathy and high sympathy), was performed for the selectivity about the clue choices. (

3) Measurement signals of NIRS
The additive average wavelength of Oxy-Hb concentration that was calculated for each experimental condition was used to calculate the arithmetic average.The probe channels from 1 through 13 were used for the data analyses excluding the channels that were interfered by noises owing to body movements and hair.
• PP wavelength + UU wavelength → congruent condition wavelength • PU wavelength + UP wavelength → incongruent condition wavelength • PP wavelength + PU wavelength → pleasant condition wavelength • UU wavelength + UP wavelength → unpleasant condition wavelength For each condition listed above, the average Oxy-Hb concentration obtained 5 to 10 s after the moment of facial expression presentation was used for calculation as the representative value.
To examine whether the amount of activity in the frontal lobe when processing incongruent emotions differs as a function of sympathy level, a two-way ANOVA, 2 levels of congruency (congruent and incongruent) × 2 levels of sympathy (low sympathy and high sympathy), was performed for the Oxy-Hb concentration under the congruent and incongruent conditions.
Similarly, to examine the amount of activity in the frontal lobe when processing pleasant and unpleasant emotions that differ as a function of sympathy level, a two-way ANOVA, 2 levels of emotion (pleasant and unpleasant) × 2 levels of sympathy (low sympathy and high sympathy), was performed for the Oxy-Hb concentration for pleasant and unpleasant conditions.( 4

) Correlational analysis
To examine the relationships among the frontal lobe activity level, developmental level, and sympathy for each experimental condition (congruent, incongruent, pleasant, and unpleasant conditions), the average Oxy-Hb concentration between 1 and 13 channels for each condition was calculated.Then, the Pearson correlations between average Oxy-Hb concentration, age, and the sympathy score for each condition were calculated.

Selection ratio of choices related to emotion estimation
Under the pleasant essay/pleasant face condition (PP condition) and the unpleasant essay/unpleasant face condition (UU condition), the emotions assumed from the essay and facial expression are congruent.Thus, the following results would be obtained.
• PP condition: happy feeling > bad feeling, others • UU condition: bad feeling > happy feeling, others Under the pleasant essay/unpleasant face condition (PU condition) and the unpleasant essay/pleasant face condition (UP condition) in which the aroused emotions are incongruent with the given context essay and the facial expression, it is often adequate to use the context as a cue to judge the emotion.Moreover, the higher the sympathy level, the less facial expressions are used to judge by intuition, and people would estimate emotion based on the context.Therefore, the following results are expected.
• "happy feeling" selection ratio under PU condition: high sympathy > low sympathy • "bad feeling" selection ratio under UP condition: high sympathy > low sympathy

Selection ratio of choices related to cues
Under the PP and UP conditions, the participants would use both context essays and facial expressions for emotion estimation.
• PP condition: both essay and facial expression > essay, facial expression • UU condition: both essay and facial expression > essay, facial expression Under the PU and UP conditions, as stated earlier, more participants in the highsympathy group than in the low sympathy group would tend to estimate the emotion from the contexts.

Selective response time for emotion estimation
The selective response time for the emotion estimation would be longer for the incongruent conditions, PU and UP conditions, than for the congruent conditions, the PP and UU conditions.
• PP and UU conditions < PU and UP conditions The response time will be shorter for the low-sympathy group under PU and UP conditions.
• Response time under PU and UP conditions: high-sympathy group < low-sympathy group

Measurement of brain function using NIRS
Previous studies (13) suggest that the prefrontal lobe is related to the processing of incongruent information.Thus, the following results will be expected.
• PP condition, UU condition < PU condition, UP condition Moreover, people with high sympathy can easily estimate emotion even under the incongruent condition aroused by the context essay and the facial expression.Thus, the prefrontal cortex will not be used for people in the low-sympathy group.
• Oxy-Hb concentration change rate under PU and UP conditions: high-sympathy group < low-sympathy group

Selection ratio of choices related to emotion estimation
Figure 3 shows the average selection ratios of each choice related to emotion estimation for each condition.For each condition, a two-way ANOVA, 3 types of choice (happy feeling, bad feeling, and others) × 2 levels of sympathy (high sympathy and low sympathy), was performed using the selection ratios, and a significant difference was found.

Selection ratio of choices related to cues
Figure 4 shows the participants' average selection ratio of each choice presented as the cues that they used to estimate the emotions for each condition.For each condition, a two-way ANOVA, 3 types of choice (essays, facial expressions, and both essay types and facial expressions) × 2 levels of sympathy (high sympathy and low sympathy), was performed for the average selection ratio.The following results were significant.

Selective response time for emotion estimation
Figure 5 shows the average response time for the emotion estimation for each condition; a two-way ANOVA, 4 experimental conditions (PP, UU, PU, and UP conditions) × 2 levels of sympathy (high sympathy and low sympathy), was performed for the average response time for each experimental condition.The following result was significant.
• Low-sympathy group: PU and UP conditions > PP condition [F(3,30) = 2.3, p < 0.01] This result shows that it is difficult for children with low sympathy to process incongruent information provided by the context essay and facial expression.

Measurement of brain function using NIRS
Figure 6 shows the average wavelength and the representative values of the average Oxy-Hb concentration at 11 channels for each experimental condition (congruent, incongruent, pleasant, and unpleasant) and sympathy group.In the figure, the change in Oxy-Hb concentration in the high-sympathy group is larger than that in the lowsympathy group.A two-way ANOVA, 2 levels of congruency (congruent and incongruent) × 2 levels of sympathy (high sympathy and low sympathy), and another two-way ANOVA, 2 emotional states (pleasant and unpleasant) × 2 levels of sympathy (high sympathy and low sympathy), were performed.However, the results of these analyses show no statistically significant main effects and interactions.

Correlational analysis
To examine the relationship between age, sympathy, and blood flow level of the prefrontal lobe, Pearson correlations were calculated using the average Oxy- Hb concentration at the prefrontal lobe for each condition (congruent, incongruent, pleasant, and unpleasant conditions), sympathy score, and age as variables.Table 2 shows correlations between variables.The result of the correlation analyses indicated significant relationships between the average Oxy-Hb concentration and the congruent condition, and between the average Oxy-Hb concentration and the pleasant condition (r = 0.82, p < 0.01).Furthermore, significant correlations were found between the average Oxy-Hb concentration and the unpleasant and pleasant conditions (r = 0.55, p < 0.05).It is difficult to interpret these significant correlations; thus, it is required to confirm that the same result would be obtained when the same study as the present study is performed.By doing so, it could be clarified that the obtained correlations in the present study are fake correlations or not.

Conclusions
In the present study, we presented the contexts and facial expressions of fictional characters to children, provided the emotion estimation tasks in which the emotions aroused by these cues are congruent or incongruent, and examined whether these tasks were performed differently depending on the individual characteristics represented by the level of sympathy.At the same time, we examined whether prefrontal lobe activities that are related to the emotion estimation differ as a function of sympathy level using NIRS.
The present study provided clues to the estimation of multiple emotions such as facial expressions and contexts that potentially become the sources for developing a human interphase related to the development of the social cognitive ability.However, more detailed experiments are expected to be performed in a future study, and an examination of the physiological index using brain waves not only by NIRS but also by other techniques and the development of brain machine interface (BMI) using these findings are also anticipated.

Figure 2
Figure2shows a flow of the experimental task.(1)The context essay was presented at the center of the computer screen (5 s).(2) Next, the facial expression picture was presented (mentioned in Step 1) in the middle of the screen (2 s).(3) A question asking to estimate the emotion of the character was presented on top of the screen, and below the question, a choice of three responses was presented (happy feeling, bad feeling, and others).The participants chose the best response from the three choices.At the bottom of the screen, another question was presented asking which cue the children used to estimate the emotion.Three choices (essays, facial expressions, and both facial expressions and essays) were given to choose the best answer.To answer these questions, the participants were required to accurately hold a numeric key corresponding to each choice for more than 3 s.(4) The processes from Steps 1 through 3 were repeated.One block consisted of 10 trials (approximately 6 min), and 8 blocks (approximately 50 min) of the tasks were conducted.The experimental condition was presented in a random order (Table1).

Fig. 1 .
Fig. 1. (Color online) Location of probe holders and channel number.

Fig. 4 .
Fig. 4. Participants' average selection ratio of cues used in the emotion estimation by group and condition.

Fig. 5 .
Fig. 5. Average response time by condition for each sympathy group (error bars indicate standard deviations).

Table 1
Experimental conditions of the tasks.(Numbers in parentheses indicate trial numbers.) Fig. 3. Average selection ratios of emotion estimation by group and condition.

Table 2 Correlations
Low-sympathy groupHigh-sympathy group Fig.6.Representative values and average wavelength of average Oxy-Hb concentration by sympathy group and condition.between average Oxy-Hb concentration for each condition, sympathy score, and age.