An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode

Abstract AlphaCode, the code generation system by DeepMind, is an AI technology for assisting software developers in solving competitive programming problems using natural language problem descriptions. However, there is no existing work comparing AlphaCode-generated codes with human codes. In this paper, we conduct an empirical study to find code similarities and performance differences between AlphaCode-generated codes and human codes. We collect 44 generated codes in C++ and Python languages from AlphaCode official website that solve 22 problems on Codeforces. We then retrieve 31,736 human codes by using Codeforces API. The results show that (i) the generated code from AlphaCode and the human code is not similar (i.e., the similarity of 0.26 for both C++ and Python), however, the code fragments in the generated code are comprised of various human codes (i.e., uniqueness of 3.30\% and 8.94\% for C++ and Python respectively) and (ii) the generated code performs on par with or worse than the human code in terms of execution time and memory usage. Moreover, AlphaCode employs excessive nested loops and unnecessary variable declarations (e.g., used \textit{long long} instead of \textit{int}), which causes a low performance regarding our manual investigation