The AI Index Annual Report is a 500-page comprehensive monster of data and charts covering AI trends in technical, public, governance, geopolitical, scientific, economic, commercial, educational, and ethical areas. Released on 15 April 2024, the Report’s conclusion is that the U.S. AI industry (not yet the U.S. government) continued to dominate AI trends in the past year and many numbers went up. The cost of the technology went up, AI system capabilities went up (in things like ‘autonomy’, video, audio, and robotics), investment went up, ancillary scientific outputs went up, the global number of AI regulations went up, and global public concern went up. Effectively every metric went up. AI is hot!
Questions
But what does all of that mean? Is AI hype or transformative change? What indications or evaluations speak to whether any of the AI trends will continue?
The answer to this question will guide strategic thinking in personal, professional, and legal areas for years to come. According to the Report, AI systems are rapidly outmatching humans in a handful of interesting areas. If we ever come to believe that AI might outmatch humans in all areas, we will seriously need to start rethinking the way we work and the kind of work we do.
I’ll consider the question of which indicators and evaluations are important over the next three articles.
This article is the first. In it, I’ll cover the major uninformative and misleading indicators in the Report. It’s useful to be able to dismiss these metrics as we encounter them again and again from AI enthusiasts and the media. The Report has nine chapters, of which I’ll advocate disregarding four: Science and Medicine, Education, Policy and Governance, and Diversity. Similarly, only a portion of the chapters on R&D, the Economy, and Public Opinion are relevant to our questions.
The second article will consider human benchmarking, looking at the useful parts of the Report’s chapters on the Economy and Public Opinion, and adding some parts from the chapter on Technical Performance.
The third article will consider equally useful machine thresholds, looking at portions of the Report’s chapters on R&D and Technical Performance, and considering for the first time the chapter on Responsible AI.
AI Patents & Publications
Much of the misleading indicators of AI progress will give rise to a fear of missing out. Perhaps you already feel it. Perhaps you’re scowling as you fight it. Regardless of whether AI is a future-defining opportunity, or not, a surge in attention will generate more attention. All of the indicators we will consider in this article will have the character of being roughly equally likely if AI is about to peak and crash, or change the world.
In Chapter 1 the Report touches on total aggregated patent applications. There was a significant surge of 62.7% more in global AI patent grants from 2021 to 2022. This escalation is part of a broader trend, with the number of granted AI patents multiplying over 31-fold since 2010. Geographically, China generates by far the most patents. In 2022, China accounted for 61.1% of all AI patent origins, starkly surpassing the United States, which held only 20.9%. This marks a significant shift from 2010, when the United States led with 54.1% of AI patents.
Chapter 1 also documents academic contributions to AI, noting that the total number of AI publications has nearly tripled over the past twelve years. Starting from approximately 88,000 publications in 2010, the figure rose to more than 240,000 by 2022. Although the past year saw a more modest increase of 1.1%, the consistent rise in publications underscores a sustained global interest and investment in AI research and development.
Significant patent activity and academic output is exciting, but without digging into these contributions on a case-by-case basis, the aggregation of these numbers is largely meaningless. Appearing at the early stages of a gold rush, many patents and papers may not lead to viable products or even may be defensively filed or written to block competitors without intending to develop on the ideas.
AI Investments, Projects & Company Adoption
Still in Chapter 1, the Report turns to projects initiated by software developers. GitHub is the most widely used web-based platform for individual developers, open-source projects, and companies to manage their development projects. The open-source AI community on GitHub has witnessed some massive growth in the past decade. Since 2011, the number of AI-related projects has increased from 845 to approximately 1.8 million by 2023. This surge included a 59.3% increase in 2023 alone. Additionally, the engagement with these projects, as measured by the total number of stars, went from 4.0 million stars in 2022 to 12.2 million in 2023.
In Chapter 4, on the economy, the Report turns to private investment in AI. In 2023, the United States further solidified its leadership, with AI investments reaching $67.2 billion, almost 8.7 times more than that of China, its nearest competitor. Despite a global downturn, where private AI investment in China and the European Union (including the UK) declined by 44.2% and 14.1% respectively, the United States saw an increase of 22.1% year over year. Conversely, global private AI investment has generally declined for the second consecutive year, though the number of newly funded AI companies increased by 40.6%, with 1,812 new startups. Additionally, the influence of AI in corporate America became more pronounced, with AI mentioned in nearly 80% of Fortune 500 companies' earnings calls in 2023, and generative AI emerging as the most frequently discussed theme.
In Chapter 5, the Report examines the impact of AI on science and medicine. In 2023, AI applications like AlphaDev and GNoME facilitated significant advancements in algorithmic sorting and materials discovery, respectively. In the medical field, AI-powered systems like EVEscape, which enhances pandemic prediction, and AlphaMissence, aiding in mutation classification, were launched. The FDA's approval of AI-related medical devices has similarly skyrocketed, with a 12.1% increase from 2021 to 2022 alone, and a more than 45-fold increase since 2012.
Sustained high numbers of well-funded AI projects don't inherently reflect the quality or transformative potential of the activities. It almost goes without saying, but the sharp increase in discussion of AI, especially in corporate settings like investor rolls and earnings calls, might reflect AI being a "buzzword" rather than an indication of genuine, effective implementation. Companies might mention AI to align with investor expectations and market trends, rather than reporting on substantial value from of AI technologies
This doesn’t mean there’s no gold in a gold rush. Increases in AI-related medical devices approved by the FDA and the development of the named AI applications in medicine do suggest some practical applications and potential value is coming online. However, which of these named products specifically prove the premise of future-defining, transformative AI? Presented as aggregate numbers and lists of promising applications, these indicators are, at best, ambiguously positive.
Public Choice: AI Opinions and AI Careers
The dynamics of the job market, educational trends, and public perceptions regarding AI are highlighted in various chapters of the Report.
Chapter 4, in the course of covering the economy, touches on employment trends. In 2023, AI-related job postings in the United States comprised only 1.6% of all job listings, down from 2.0% the previous year. Tech workers generally had a bad year in 2023. This decline can be attributed to a reduction in the number of vacancies advertised by leading AI firms, along with the decreased proportion of tech roles within these companies. This cooling in the AI job market does contrast with the broader enthusiasm for AI.
Chapter 6 describes educational trends in computer science (CS), as the educational pipeline feeding the AI industry. Internationally, the proliferation of AI-focused academic programs is evident. Since 2017, the number of English-language, AI-related postsecondary degree programs have tripled. However, North America shows mixed trends, perhaps in step with the cooling of the AI job market. The number of American and Canadian bachelor’s graduates in CS has seen a consistent increase over more than a decade. However, the growth in new graduates at the master's and PhD levels has flattened since 2018, with a modest uptick in PhD graduates. Despite that trend, the migration of AI PhDs to industry roles has accelerated: in 2022, 70.7% of new AI PhDs opted for industry positions, up from 40.9% in 2011, illustrating a significant brain drain from academia to industry. In secondary education, the number of American high school students taking AP computer science exams has surged, with 201,000 exams administered in 2022, a more than tenfold increase since 2007.
Chapter 9 deals with public awareness and apprehension about AI. An Ipsos survey highlights that the proportion of people who believe AI will significantly impact their lives in the next three to five years rose from 60% to 66% over the last year. Additionally, nervousness towards AI products and services has grown, with 52% of respondents expressing concern, a significant increase of 13 points from the previous year. This rising apprehension reflects growing societal engagement with both the potential and risks associated with AI.
These insights reflect structural and strategic shifts in employment, academia, and societal engagement with AI technologies. There is more information revealed by these collective changes of life paths than in board call or patent filings. Decisions about what to study in university are clearly not mere surface-level enthusiasm. However, it is, as with the previous indicators, equally consistent with large parts of society succumbing to a bandwagon effect.
AI Laws
Regulators are also feeling compelled to act on AI in large numbers.
Chapter 7 in the Report touches on the growth in AI-related laws, amid discussions of policy and governance more broadly. In the United States, the regulatory framework surrounding AI has grown, primarily at the State level. In 2023, the number of AI-related regulations reached 25, a significant increase from just one regulation in 2016. This is a compound annual growth rate of 56.3% in AI regulations over the past year. Internationally, the discourse on AI has intensified within legislative bodies. Mentions of AI in legislative proceedings around the world nearly doubled, rising from 1,247 mentions in 2022 to 2,175 in 2023. AI was discussed in the legislative assemblies of 49 countries last year, with at least one country from every continent represented.
Increasing regulation, as with the previous indicators, is not indicative of transformative technologies. Regulation can be both preventive, aiming to set frameworks before problems become widespread, and reactive, responding to public outcries or high-profile incidents. The challenge is to go one step deeper in the analysis to determine the indications that the predictions and public outcries generating regulations, job postings, educational programs, or corporate discussions are, themselves, well founded.
That’s what we’ll do in part 2 and 3.