Monitoring emotions and cooperative behavior

Cooperation among people in teams that are bound to perform a common goal is one of the main factors determining success of these teams. Cooperation becomes even more important for small teams performing long-term missions in isolation. Examples of such missions include missions performed on the international space stations, polar research stations, submarines, oil platforms and meteorological stations. Such missions are usually performed in extreme physical and psychological conditions that have a strong negative effect on emotional states of the crew members and interpersonal relationships between them. This work presents a set of methods for an automatic measuring of cooperative behavior and emotional states in goal-oriented teams performing long-term missions in isolation. The proposed methods were tested on and applied to data from the Mars-500 isolation experiment. The methods are based on analysis of two kinds of data: behavior in a cooperative computer game and video records of facial expressions. Chapter 2 introduces the Colored Trails (CT) game that is used as a research test-bed for investigating decision-making in groups comprising people, computer agents, and a mix of these two. Chapter 2 also proposes to use the CT game with specially designed payoff spaces that make it easier to tie decisions of the players in the game with the social preferences of the players. Chapter 3 introduces a numerical parameter describing the propositions made in the CT game in a way that is relevant to cooperative and fair behavior of players. This parameter has been used to demonstrate that different players have different social preferences. To describe cooperative behavior of players in the proposal phase of the game two ratios were introduced in Chapter 4. The first ratio, called level of cooperation, is used to describe preferences of each player over cooperation and fairness. The second ratio, called symmetry of cooperation, describes difference between sacrificing and exploiting behavior of players. The introduced ratios provide a quantitative way to describe cooperative behavior of players in the game. Chapter 4 demonstrates that different players behave differently in terms of the introduced ratios. Moreover, the behavior of a player can depend on who his/her opponent is in the game. It indicates that cooperative games can be used to capture some cooperative aspects of interpersonal relationships between different players. Chapter 5 proposes a utility function that describes cooperation and different kinds of fairness. To derive this utility function, Chapter 5 starts from building a link between the social-welfare and inequity-aversion utility functions that are well established in literature. Moreover, the way in which these functions are generalized to the case of more-than-two subjects is considered and an alternative generalization procedure is proposed. Finally, the utility function is extended by adding a term that models an additional type of fairness. For the case of two players, the introduced utility function contains three parameters describing the players' social preferences. The utility-based approach can be considered as a more general than the approach based on the use of cooperation ratios because the two cooperation ratios can be obtained from the values of the three utility parameters but the opposite is not true. Chapter 6 considers influence of the combinatorial complexity of the game on the decisions of the players to develop a model that can explain non optimal decisions of the players in the game. Three parameters that determine combinatorial complexity of the decisions in the game were identified and their influence on the decisions of each player has been demonstrated. To find correct values of the utility parameters, describing social preferences of a player, it is necessary to take into account the fact that the ability of players to find options in the game is limited by the combinatorial complexity of the game. To do that, different measures of inconsistency between a given set of utility parameters and a given set of decisions were proposed. It is demonstrated that a correct choice of a measure of inconsistency is very important for the proper estimation of the utility parameters. Moreover, a model-based approach to the construction of the inconsistency measures was proposed and it was demonstrated that this approach provides a better way to estimate the utility parameters of the players. Chapter 7 presents a method for description of the social preferences of the players revealed in the response phase by a network of player's preferences over different types of proposals. Different ways to quantify these preferences were proposed. A link between the methods, developed for the proposal and response phases of the game has been made and, in this way, it has been demonstrated that the parameters calculated for the different phases and describing social preferences of the players are in a statistically significant agreement. Finally, Chapters 8 and 9 are devoted to analysis of video records of facial expressions taken during the experiment. For the analysis of the facial expressions different statistical properties were introduced, their classification was given and their meaning was described. By considering these properties two interesting results were found. First, there is a statistically significant correlation between the emotional states of the crew for the neighboring experiments (separated by two weeks). Second, different types of the relations between the facial expressions of different users were found. A possibility to use these relations as quantitative measures of emotional bonding of different types and strengths was discussed. In addition to the use of the statistical properties a mathematical model that could help to interpret time-dependent facial expressions in terms of events inducing emotions was developed. With the proposed method the locations of these events in time as well as the type and value of the impact that these events have on emotional states can be found. In summary, a methodological toolbox for an automatic monitoring of the psychosocial atmosphere in small groups performing long-term missions in isolation is proposed. The possibility to use computer games to monitor interaction between crew members and, in this way, access some aspects of their interpersonal relations was studied. Special attention has been paid to cooperative aspects of interpersonal relations. To monitor emotional states of the crew members commercial FaceReader software has been used. The works focuses on the method of analysis of facial expressions, which are relevant to the monitoring of interpersonal relations and long-term effects of isolation. This work can be considered as a first step in the development of systems for an automatic monitoring of psychosocial atmosphere in goal-oriented teams. A lot of additional research and testing has to be done before a comprehensive system of this kind can be created. Further progress in this area would involve the following: First, the games used for a monitoring of interactions between the crew members have to be made gradually richer in terms of the decision making situations to capture different aspects of interpersonal relations. Any extension of the games has to be supported by a development of theories for analysis and interpretation of the playing behavior in the extended settings. Second, the combinatorial complexity of the game has to be decreased to decrease the time needed for making decisions and, in this way, increase the number of decisions (data for analysis) per game. Moreover, reduction of the combinatorial complexity can help to separate effects of social preferences from the effects of complexity. For a more comprehensive monitoring of emotional states, measurements of voice intonations look very promising since humans are shown to express their emotions through facial expression and voice intonation. Moreover, usually people express their emotions through voice intonation during their communication with each other. This aspect can be very interesting in the context of monitoring interpersonal relations. Finally, a progress in the development of tools for an automatic facial expression recognition and voice intonation interpretation can be accomplished by the development of methods for analysis of the data generated by these tools. In other words, we need models that can bind the observed (and classified) facial expressions and voice intonations with underlying psychological processes in a deeper way