Pre-test session impact on the effectiveness assessment of a fire safety game

In recent years, critiques have been formulated regarding current evaluation methods of DGBL (digital game-based learning) effectiveness, putting the validity of certain results in doubt. An important point of discussion in DGBL effectiveness studies is whether or not a pre-test should be administered, as it can lead to practice effects and pre-test sensitization, threatening internal validity of the results. The present study aims at testing if the administration of a pre-test has a direct influence on post-test scores and/or makes participants more receptive to the intervention. For this purpose, an effectiveness study of a fire safety training in a hospital was conducted using a Solomon four-group design. The experimental groups received a game-based intervention (n= 65) of which 34 participants received a pre-test and 31 did not. The control groups received traditional classroom instruction (n=68), of which 39 participants received a pre-test and 29 did not. A 2x2 ANOVA was used to explore the practice effect and the interaction between the pre-test and the intervention. An interaction effect between pre-test and intervention is detected. More specifically, this interaction takes place in the traditional classroom group, indicating pre-test sensitization. In the traditional classroom context, the pre-test makes the participants more sensitive to the content treated in the intervention while administration of a pre-test does not influence outcomes of the DGBL treatment. When the administration of a pre-test influences the control group's receptivity to the treatment, but not the experimental group, results of an effectiveness study may be biased. This is especially relevant in the DGBL field as often, non-significant differences between DGBL and more traditional methods are reported. Therefore, further research should take this into account and look for possible solutions to solve this discrepancy.In recent years, critiques have been formulated regarding current evaluation methods of DGBL (digital game-based learning) effectiveness, putting the validity of certain results in doubt. An important point of discussion in DGBL effectiveness studies is whether or not a pre-test should be administered, as it can lead to practice effects and pre-test sensitization, threatening internal validity of the results. The present study aims at testing if the administration of a pre-test has a direct influence on post-test scores and/or makes participants more receptive to the intervention. For this purpose, an effectiveness study of a fire safety training in a hospital was conducted using a Solomon four-group design. The experimental groups received a game-based intervention (n= 65) of which 34 participants received a pre-test and 31 did not. The control groups received traditional classroom instruction (n=68), of which 39 participants received a pre-test and 29 did not. A 2x2 ANOVA was used to explore the practice effect and the interaction between the pre-test and the intervention. An interaction effect between pre-test and intervention is detected. More specifically, this interaction takes place in the traditional classroom group, indicating pre-test sensitization. In the traditional classroom context, the pre-test makes the participants more sensitive to the content treated in the intervention while administration of a pre-test does not influence outcomes of the DGBL treatment. When the administration of a pre-test influences the control group's receptivity to the treatment, but not the experimental group, results of an effectiveness study may be biased. This is especially relevant in the DGBL field as often, non-significant differences between DGBL and more traditional methods are reported. Therefore, further research should take this into account and look for possible solutions to solve this discrepancy.P