General intelligence disentangled via a generality metric for natural and artificial intelligence

[EN] Success in all sorts of situations is the most classical interpretation of general intelligence. Under limited resources, however, the capability of an agent must necessarily be limited too, and generality needs to be understood as comprehensive performance up to a level of difficulty. The degree of generality then refers to the way an agent's capability is distributed as a function of task difficulty. This dissects the notion of general intelligence into two non-populational measures, generality and capability, which we apply to individuals and groups of humans, other animals and AI systems, on several cognitive and perceptual tests. Our results indicate that generality and capability can decouple at the individual level: very specialised agents can show high capability and vice versa. The metrics also decouple at the population level, and we rarely see diminishing returns in generality for those groups of high capability. We relate the individual measure of generality to traditional notions of general intelligence and cognitive efficiency in humans, collectives, non-human animals and machines. The choice of the difficulty function now plays a prominent role in this new conception of generality, which brings a quantitative tool for shedding light on long-standing questions about the evolution of general intelligence and the evaluation of progress in Artificial General Intelligence.We are grateful to Jim DiCarlo and his lab (http://dicarlolab.mit.edu/, MIT), Judith Burkart, Laura Damerius and Carel van Schaik (Universitat Zurich), and Katherine Bruce and Mark Galizio (UNC Wilmington), for their data and the very helpful discussions. We thank David Stillwell for suggesting the relation between person-fit metrics and an early version of generality, and Heinrich Peters for pointing out Guttman's model. JHO has been partially supported by the EU (FEDER) and Spanish MINECO grant RTI2018-094403-B-C32 funded by MCIN/AEI/10.13039/501100011033 and by "ERDF A way of making Europe", Generalitat Valenciana under grant PROMETEO/2019/098 and EU's Horizon 2020 research and innovation programme under grant agreement No. 952215 (TAILOR). JHO and SOH are funded by the Future of Life Institute, FLI, under grant RFP2-152. JHO and LC are funded by US DARPA HR00112120007 (RECoG-AI). LC and SOH are funded by the Leverhulme Trust through a grant for the Leverhulme Centre for the Future of Intelligence. FMP is funded by the AI-Watch and HUMAINT projects by DG CONNECT and DG JRC of the European Commission. All authors declare no competing interests. We thank John Burden, Richard Evans and the anonymous reviewers for their insightful comments and corrections. Code and data are available for reproducibility at http://github.com/jorallo/gener ality as explained in G.Hernández-Orallo, J.; Loe, BS.; Cheke, L.; Martínez-Plumed, F.; Heigeartaigh, SO. (2021). General intelligence disentangled via a generality metric for natural and artificial intelligence. Scientific Reports. 11(1):1-16. https://doi.org/10.1038/s41598-021-01997-7116111Pintner, R. Intelligence and its measurement: A symposium-v. J. Educ. Psychol. 12, 139 (1921).Fogel, D. B. Evolutionary Computation: Toward a New Philosophy of Machine Intelligence (IEEE Press, 1995).Legg, S. & Hutter, M. Universal intelligence: A definition of machine intelligence. Minds Mach. 17, 391–444 (2007).Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).Yang, J. et al. General learning ability in perceptual learning. Proc. Natl. Acad. Sci. 117, 19092–19100 (2020).Spearman, C. The Abilities of Man: Their Nature and Measurement (Macmillan, 1927).Detterman, D. K. General intelligence: Cognitive and biological explanations. In The General Factor of Intelligence: How General is it? (eds Sternberg, R. J. & Grigorenko, E. L.) 223–243 (Lawrence Erlbaum Associates, 2002).Burkart, J. M., Schubiger, M. N. & van Schaik, C. P. The evolution of general intelligence. Behav. Brain Sci. 40, e195 (2017).Poirier, M.-A., Kozlovsky, D. Y., Morand-Ferron, J. & Careau, V. How general is cognitive ability in non-human animals? A meta-analytical and multi-level reanalysis approach. Proc. R. Soc. B 287, 20201853 (2020).Newell, A., Shaw, J. C. & Simon, H. A. Report on a general problem-solving program. In IFIP Congress, 256–264 (1959).McCarthy, J. Generality in artificial intelligence. Commun. ACM 30, 1030–1035 (1987).Johnson, W. Correlation and explaining variance: To square or not to square?. Intelligence 39, 249–254 (2011).Detterman, D. K. & Daniel, M. H. Correlations of mental tests with each other and with cognitive variables are highest for low IQ groups. Intelligence 13, 349–359 (1989).Hernández-Orallo, J. The Measure of All Minds: Evaluating Natural and Artificial Intelligence (Cambridge University Press, 2017).Hernández-Orallo, J. et al. A new AI evaluation cosmos: Ready to play the game? AI Magazine 38 (2017).Hernández-Orallo, J. Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artif. Intell. Rev. 48, 397–447 (2017).Lapuschkin, S. et al. Unmasking clever Hans predictors and assessing what machines really learn. Nat. Commun. 10, 1096 (2019).Hernández-Orallo, J. Gazing into Clever Hans machines. Nat. Mach. Intell. 1 (2019).Cave, S. & ÓhÉigeartaigh, S. S. Bridging near-and long-term concerns about AI. Nat. Mach. Intell. 1, 5–6 (2019).Hernández-Orallo, J. Unbridled mental power. Nat. Phys. 15, 106 (2019).Sternberg, R. J. The holey grail of general intelligence. Science 289, 399–401 (2000).Carroll, J. B. The Three-Stratum Theory of Cognitive Abilities (The Guilford Press, 1997).Sternberg, R. A triarchic view of giftedness: Theory and practice. Handbook of Gifted Education 2, 43–53 (1997).Rust, J., Kosinski, M. & Stillwell, D. Modern Psychometrics: The Science of Psychological Assessment 4th edn. (Routledge, 2020).Shaw, R. C. & Schmelz, M. Cognitive test batteries in animal cognition research: Evaluating the past, present and future of comparative psychometrics. Anim. Cogn. 20, 1003–1018 (2017).Ramón, S. et al. Textura del sistema nervioso del hombre y de los vertebrados: estudios sobre el plan estructural y composición histológica de los centros nerviosos adicionados de consideraciones fisiológicas fundadas en los nuevos descubrimientos, vol. 1 (Moya, 1899).Del Giudice, M. & Crespi, B. J. Basic functional trade-offs in cognition: An integrative framework. Cognition 179, 56–70 (2018).Nielsen, T. S., Barros, G. A., Togelius, J. & Nelson, M. J. Towards generating arcade game rules with VGDL. In 2015 IEEE Conference on Computational Intelligence and Games (CIG), 185–192 (IEEE, 2015).Insa-Cabrera, J., Dowe, D. L., España-Cubillo, S., Hernández-Lloreda, M. V. & Hernández-Orallo, J. Comparing humans and AI agents. In International Conference on Artificial General Intelligence, 122–132 (Springer, 2011).Elithorn, A. A preliminary report on a perceptual maze test sensitive to brain damage. J. Neurol. Neurosurg. Psychiatry 18, 287 (1955).Loe, B. S. & Rust, J. The perceptual maze test revisited: Evaluating the difficulty of automatically generated mazes. Assessment 26, 1524–1539 (2019).Davies, A. D. & Davies, M. The difficulty and graded scoring of Elithorn’s perceptual maze test. Br. J. Psychol. 56, 295–302 (1965).Thurstone, L. L. Primary mental abilities. Psychometric Monographs (1938).Simon, H. A. & Kotovsky, K. Human acquisition of concepts for sequential patterns. Psychol. Rev. 70, 534 (1963).Hernández-Orallo, J., Martínez-Plumed, F., Schmid, U., Siebers, M. & Dowe, D. L. Computer models solving intelligence test problems: Progress and implications. Artif. Intell. 230, 74–107 (2016).Hernández-Orallo, J. Beyond the turing test. J. Logic Lang. Inf. 9, 447–466 (2000).Katayama, S. An analytical inductive functional programming system that avoids unintended programs. In Proceedings of the ACM SIGPLAN 2012 Workshop on Partial Evaluation and Program Manipulation, 43–52 (ACM, 2012).Rajalingham, R. et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 38, 7255–7269 (2018).Young, J. W. et al. The odour span task: A novel paradigm for assessing working memory in mice. Neuropharmacology 52, 634–645 (2007).April, L. B., Bruce, K. & Galizio, M. The magic number 70 (plus or minus 20): Variables determining performance in the rodent odor span task. Learn. Motiv. 44, 143–158 (2013).Smith, M. R., Martinez, T. & Giraud-Carrier, C. An instance level analysis of data complexity. Mach. Learn. 95, 225–256 (2014).Martínez-Plumed, F., Prudêncio, R. B., Martínez-Usó, A. & Hernández-Orallo, J. Item response theory in AI: Analysing machine learning classifiers at the instance level. Artif. Intell. 271, 18–42 (2019).Morizot, J., Ainsworth, A. T. & Reise, S. P. Toward modern psychometrics. Handbook of Research Methods in Personality Psychology 407 (2009).Herrmann, E., Call, J., Hernández-Lloreda, M. V., Hare, B. & Tomasello, M. Humans have evolved specialized skills of social cognition: The cultural intelligence hypothesis. Science 317, 1360–1366 (2007).Koorehdavoudi, H. & Bogdan, P. A statistical physics characterization of the complex systems dynamics: Quantifying complexity from spatio-temporal interactions. Sci. Rep. 6, 1–13 (2016).Balaban, V., Lim, S., Gupta, G., Boedicker, J. & Bogdan, P. Quantifying emergence and self-organisation of Enterobacter cloacae microbial communities. Sci. Rep. 8, 1–9 (2018).Hazy, J. K. When the value of cooperating scales: Collective intelligence, information processing, and the leadership meta-capability. Front. Hum. Dyn. 3, 1 (2021).Seed, A., Emery, N. & Clayton, N. Intelligence in corvids and apes: A case of convergent evolution?. Ethology 115, 401–420 (2009).Jacobs, I. & Gärdenfors, P. The false dichotomy of domain-specific versus domain-general cognition. Behav. Brain Sci. 40, e207 (2017).Brynjolfsson, E. & Mitchell, T. What can machine learning do? Workforce implications. Science 358, 1530–1534 (2017).Brown, N. & Sandholm, T. Superhuman AI for multiplayer poker. Science 365, 885–890 (2019).Rahwan, I. et al. Machine behaviour. Nature 568, 477–486 (2019).Fleuret, F. et al. Comparing machines and humans on a visual categorization test. Proc. Natl. Acad. Sci. 108, 17621–17625 (2011).Ullman, S., Assif, L., Fetaya, E. & Harari, D. Atoms of recognition in human and computer vision. Proc. Natl. Acad. Sci. 113, 2744–2749 (2016).Firestone, C. Performance vs competence in human–machine comparisons. Proc. Natl. Acad. Sci. 117, 26562–26571 (2020).Crosby, M. et al. The animal-AI testbed and competition. Proceedings of Machine Learning Research 164–176 (2020).Shanahan, M., Crosby, M., Beyret, B. & Cheke, L. Artificial intelligence and the common sense of animals. Trends Cogn. Sci. 24, 862–872 (2020).Tenenbaum, J. B. & Griffiths, T. L. Generalization, similarity, and Bayesian inference. Behav. Brain Sci. 24, 629 (2001).Balla, M., Lucas, S. M. & Perez-Liebana, D. Evaluating generalisation in general video game playing. In 2020 IEEE Conference on Games (CoG), 423–430 (2020).Risi, S. & Togelius, J. Increasing generality in machine learning through procedural content generation. Nat. Mach. Intell. 2, 428–436 (2020).Schrittwieser, J. et al. Mastering Atari, go, chess and shogi by planning with a learned model. Nature 588, 604–609 (2020).Brown, T. B. et al. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), arXiv preprintarXiv:2005.14165 (2020)