During the National Day, two researchers from the University of Cambridge, Nathan Benaich and Ian Hogarth, launched the annual 2020 State of AI Report (State of AI Report).
Similar to previous years, this report still summarizes the new technological advances in the AI field in the past year, new changes in the industrial structure, and new features of government policies from the aspects of industry, talents, policies, and forecasts. Make predictions about the future.
AI research and technology: only 15% of the papers will open the code, PyTorch is more popular than TensorFlow
AI research is not as open as expected. Only 15% of papers open source their code.
The release of the research paper code is critical to AI-related accountability, reproducibility, and progress.
Since mid-2016, this indicator has hardly improved. Generally speaking, academic groups are more likely to release their code than industry groups. Well-known organizations that have not released all the code include OpenAl and DeepMind. For large technology companies, their code is often intertwined with their private infrastructure.
Facebook’s PyTorch is rapidly surpassing Google’s TensorFlow in the use of research papers.
In the conference papers (20-35%) that mentioned the use of frameworks, 75% cited PyTorch, but did not mention TensorFIow. At the same time, the author observes that TensorFlow, Caffe and Caffe2 are still the main force in AI research.
In the implementation of the paper on GitHub, PyTorch is also more popular than TensorFlow. Among them, 47% of the papers are based on PyTorch, and TensorFIow is 18%. PyTorch provides greater flexibility and dynamic calculation graphs, making experiments easier. JAX is a framework produced by Google that is more math-friendly, and is usually favored in work other than convolutional models and transformers.
NLP model: one billion parameter club
The report analyzes the trend of the most popular NLP models today.
For NLP models, performance improvement obviously requires larger models, data sets and higher computational budgets.
According to data released by Google, every 1,000 parameters cost an average of $1. This means that OpenAI training GPT-3 with 175 billion parameters may cost tens of millions of training costs. Experts speculate that the possible budget will exceed $10 million.
The report pointed out that without a major breakthrough, it would take hundreds of billions of dollars to reduce the ImageNet error rate from 11.5% to 1%! Many practitioners believe that progress in the mature field of ML is currently somewhat stagnant.
The new generation of transformer language models are unlocking new NLP use cases. GPT-3, T5, and BART are greatly improving the performance of transformer models for text-to-text tasks (such as translation, text summarization, text generation, text-to-code).
In addition, in 2019, the new NLP benchmark SuperGLUE was officially released. More than a dozen teams surpassed humans in the GLUE benchmark test.
What will be after SuperGLUE?
The multi-task language comprehension challenge test covers 57 tasks, including mathematics, American history, law, etc., to comprehensively test world knowledge and problem-solving skills. There is still a huge knowledge gap in GPT-3.
Biology’s “AI Moment”
Biological research is experiencing an “AI moment”: in 2020 alone, more than 21,000 related papers will be published. Since 2017, publications involving AI methods (such as deep learning, NLP, computer vision, RL) have increased by more than 50% year-on-year.
Large labeled data sets bring great potential and enrich new biological knowledge related to health and disease.
From physical object recognition to “cell painting”: Biology through image decoding
Federal Learning
Federal Learning was initiated by Google in 2016 and is currently booming. From 2018 to 2019, the number of papers mentioning federated learning has increased almost five times. The number of related papers published in the first half of 2020 exceeds the number of the whole year of 2019.
AI talents: Professors flow from universities to technology companies, 27% of the top AI talents in the United States come from China
In recent years, the demand for artificial intelligence professors from technology companies has increased.
From 2004 to 2018, Google, DeepMind, Amazon, and Microsoft hired 52 tenured professors from American universities. Correspondingly, 38 professors from Carnegie Mellon University, University of Washington and University of Berkeley left their posts. However, it is worth noting that no artificial intelligence professor left in 2004, and 41 artificial intelligence professors chose to leave in 2018 alone.
Of course, the departure of the old professor may free up the ladder of promotion for young academic talents. At the same time, some scholars did not buy it.
Professor’s departure caused a decline in academic graduation creative ability
Some companies, including Facebook, use the dual academic/industry cooperation mechanism to mine human resources. What impact does this have on universities?
According to the report, the loss of artificial intelligence professors is very important to colleges and universities. Among 69 universities in the United States, the decline in graduate entrepreneurship is related to the departure of professors. Generally speaking, 4-6 years after the departure of a tenured artificial intelligence professor, the probability of graduates starting an artificial intelligence company is reduced by 4%; but this does not apply to the situation where the professor leaves 1-3 years before the student graduates. This shows that the interaction between professors and students is important; but there is no significant correlation between the departure of artificial intelligence professors and the establishment of non-AI companies by graduates of the same university.
Can 100 million euros be “buy” for 50 professors?
Many colleges and universities are investing heavily in AI talent and discipline construction.
For example, the Eindhoven University of Technology (TUE) in the Netherlands has pledged to invest 100 million euros in 5 years to create a new research institute focused on the use of intelligent algorithms on machines such as robots and self-driving cars.
TUE ranked 120th in QS World University Rankings in 2021
Silver Lake (Silver Lake) founder donated US$100 million to create Roux Institute at Northeastern University, a new company focused on artificial intelligence applied to digital and life sciences. Graduate school. It will develop in the fields of applied analysis, computer science, data science, data visualization and machine learning, as well as bioinformatics, biotechnology, genomics, health data analysis and precision medicine.
In 2019, Abu Dhabi announced the establishment of Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), which is the first in the worldResearch artificial intelligence university.
Abu Dhabi established “the world’s first artificial intelligence university”
Mohammed bin Zayed AI University (MBUZAI) is a new research-oriented higher education college
MBUZAI received 2223 applications from 97 countries, most of which came from outside the Middle East and North Africa.
The United States continues to dominate the NeurIPS 2019 papers
Judging from the publication of NeurIPS 2019 papers, the United States maintains its dominant position, with Google, Stanford University, CMU, Massachusetts Institute of Technology and Microsoft Research occupying the top five.
However, 29% of the paper authors admitted to NeurIPS 2019 have obtained their undergraduate degrees in China.
But after leaving Chinese universities, 54% of graduates chose to go to the United States to publish papers in NeurIPS.
In 2019, the United States attracted more than half of foreign NeurIPS authors.
According to various data, the United States is a country with a very strong reserve of post-doctoral talents, and nearly 90% of Chinese and non-Chinese students who have obtained doctoral degrees in the United States stay in the United States to work.
By comparison, it is found that foreign graduates of the US Ph.D. Program in Artificial Intelligence are most likely to work in large companies, while native Americans are more likely to work in startups or academia.
However, after obtaining a PhD in artificial intelligence in the United States, not everyone stays in the United States. Some of them will choose to leave, and the most likely to go to the United Kingdom and China.
55% of graduates who go to the UK choose to work in the private sector; 40% of those who go to China choose to go to the private sector.
There are also data showing that despite the leading AI technology in the United States, most of the top artificial intelligence researchers working in the United States are not undergraduate education in the United States, China (27%), Europe (11%) and India ( 11%) are the largest delivery countries of AI talents in the United States.
On the whole, AI talents are in short supply.
As one of the hottest positions in the 21st century, AI-related positions can be said to be hot. From IndeAn analysis of US data from ed.com shows that the number of job offers is about three times the number of job seekers.
Although the artificial intelligence talent market is hot, it is not immune to the impact of the epidemic. On LinkedIn, public recruitment advertisements mentioning the deep learning framework have increased significantly in 2020, but since February 2020, it has also been The emergence of the epidemic has also declined.
AI industry: the medical and autonomous driving fields continue to be hot
In the AI industry, the report analyzes the importance of AI in fields such as medical care and driving.
I have to say that strong financial support is an important force in the development of the AI industry. In China, travel companies like Didi chose to spin off their autonomous driving business and raised $500 million from institutions such as the SoftBank Vision Fund. By July of this year, Didi launched its self-driving car service in Shanghai.
During the epidemic, many technology companies put AI medical image recognition technology into use.
For example, deep learning improves super-resolution microscopy imaging from acquisition to analysis, and uses supervised learning and computer vision to reduce the hours under the human microscope to minutes. Super-resolution microscopes usually require subject matter experts to evaluate samples, and ONI’s system automates these visual inspection tasks and unlocks super-resolution non-professional users.
Moreover, the U.S. Centers for Medical Insurance and Medicaid Services also proposed the cost standards for medical imaging products based on deep learning. In the future, prioritizing the use of AI technology will become more and more common in the medical field. For example, the use of artificial intelligence to design drugs has already undergone clinical trials in Japan, and a large number of startup companies have also received a lot of funds for the development of platform strategies.
In terms of autonomous driving, more than half of the states in the United States have enacted legislation on autonomous driving.
This article is only a part of the report. For more detailed information, please refer to the full report.