Statistical evaluation of the predictive toxicology challenge 2000-2001

Motivation The development of in silico models to pre-dict chemical carcinogenesis from molecular structure would help greatly to prevent environmentally caused cancers. The Predictive Toxicology Challenge (PTC) competition was organized to test the state-of-the-art in applying machine learning to form such predictive models. Results Fourteen machine learning groups generated 111 models. The use of Receiver Operating Character-istic (ROC) space allowed the models to be uniformly compared regardless of the error cost function. We de-veloped a statistical method to test if a model performs significantly better than random in ROC space. Using this test as criteria five models performed better than random guessing at a significance level of 0.05. Sta-tistically the best predictor was the Viniti model for fe-male mice, with value below 0.002. The toxicolog-ically most interesting models were Leuven2 for male mice, and Kwansei for female rats. These models per-formed well in the statistical analysis and they are in the middle of ROC space, i.e., distant from extreme cost as-sumptions. These predictive models were also indepen-dently judged by domain experts to be among the three most interesting, and are believed to include a small but significant amount of empirically learned toxicological knowledge. Availability PTC details and data can be foun