Automated testing of real-life self-driving systems and beyond

Keynote speech at the IEEE International Conference on Artificial Intelligence Testing (AI Test 2019), San Francisco East Bay, California, USA, Apr. 9, 2019. In this keynote, Dr. Z.Q. (George) Zhou, a pioneer of metamorphic testing, points out the following fallacies related to the development, testing, and application of artificial intelligence and machine learning systems: the “intelligence” fallacy, the “testers can always decide” fallacy, the “reliability” fallacy, and the “safety driver” fallacy. Dr. Zhou highlights metamorphic testing as a proven solution to many of the testing challenges of AI systems. He shows that the concept of metamorphic testing can enable test automation for self-driving vehicles, autonomous drones, as well as other types of artificial intelligence applications such as machine translations, search engines, and Google Maps. In this keynote, Dr. Zhou also analyzes the Boeing 737 Max 8 crashes in 2018 and 2019, and points out that the current solution (software fix) proposed by Boeing has not solved the problem, and that such software fix will introduce new dangers. Dr. Zhou highlights metamorphic testing as a feasible solution to such problems.Presentation slides (PDF files), showing the detection of real-life bugs in AI and self-driving systems including Baidu Apollo, Tesla, Google Maps, Bing, and so on