A tale of two interactions: Inferring performance in hospitality encounters from cross-situation social sensing

People behave differently in different situations. With the advances in ubiquitous sensing technologies, it is now easier to capture human behavior across multiple situations automatically and unobtrusively. We investigate human behavior across two situations that are ubiquitous in hospitality (job interview and reception desk) with the objective of inferring performance on the job. Utilizing a dataset of 338 dyadic interactions, played by students from a hospitality management school, we first study the connections between automatically extracted nonverbal cues, linguistic content, and various perceived variables of soft skills and performance in these two situations. A correlation analysis reveals connection between perceived variables and nonverbal cues displayed during job interviews, and perceived performance on the job. We then propose a computational framework, with nonverbal cues and linguistic style from the two interactions as features, to infer the perceived performance and soft skills in the reception desk situation as a regression task. The best inference performance, with R2 = 0.40, is achieved using a combination of nonverbal cues extracted from the reception desk setting and the human-rated interview scores. We observe that some behavioral cues (greater speaking turn duration and head nods) are positively correlated to higher ratings for all perceived variables across both situations. The best performance using verbal content is achieved by fusion of LIWC and Doc2Vec features with R2 = 0.25 for perceived performance. Our work has implications for the creation of behavioral training systems with focus on specific behaviors for hospitality students