Construct validity of situational judgment tests: An examination of the effects of agreeableness, organizational leadership culture, and experience on SJT responses

Numerous factors are likely to influence response patterns to situational judgment tests, including agreeableness, leadership style, impression management, and job and organizational experience. This research presents background information and research on situational judgment tests and several constructs hypothesized to influence situational judgment test responses. A situational judgment test and manipulations to influence response patterns were developed and piloted with a small sample of management professionals and undergraduate students. Larger samples of management professionals and undergraduate students participated in the experimental research. Participants were asked to imagine that they are applying for a job. Each participant was presented with background information about a fictitious company, describing a company as either highly Participative/Supportive or highly Directive/Achieving in its leadership culture. A third description provided no information about leadership culture to serve as a control. Participants responded to a situational judgment test consisting of some commercially developed items and some new items. Then participants responded to an inventory comprised of items that measure the factors hypothesized to influence response patterns, specifically Agreeableness and Experience. Significant differences in response patterns were determined to be attributable to the Agreeableness and Experience variables, and the Leadership Culture manipulations, as well as the interaction between Experience and the Leadership Culture manipulations. No significant differences were clearly attributable to the Agreeableness by Leadership Culture interaction. The ramifications of these findings are discussed and recommendations for future research are presented