Information distribution is an ancient problem.
Editor’s note: This article is from WeChat public account “Hedgehog Commune” (ID: ciweigongshe ), author Tong Shuting.
Algorithm distribution, editing distribution, social distribution… In the information age, people often discuss information distribution issues, and related concepts have become a hot term.
But in fact, information resources have always played an important role in the evolution and development of human beings. Social distribution is the oldest distribution method, and the editorial distribution is earlier than we thought.
And when we put algorithmic distribution in the history of human social information distribution, we can clearly see its “front” and “post” – from this point of view, fresh algorithm recommendations, in fact, not fresh.
After the information distribution problem
Information distribution is an ancient problem.
Start with an interesting association: In the early days of human civilization, the ancestors of the group depended on collecting and hunting to survive. Since hunting is a very dangerous technical activity, people need to exchange information and experience in hunting operations to increase the success rate.
For example, what signal is used to call a companion when the prey appears? From what position is the ambush beast better? They use gestures and voices to distribute this important information to their peers – this is “social distribution,” the most primitive form of information distribution in human society.
“Knowledge” (Oracle):
Oracle The “knowledge” means talking about and teaching the experience of hunting and fighting.
Social distribution means direct and natural distribution based on social relationships. “A Brief History of Humanity” uses “gossip” to describe this kind of information exchange, pointing out the important role of gossip in human evolution.
Another form of information distribution that has existed since ancient times is edited and distributed. Although the word “edit” in English appears to be related to newspapers, this form of distribution has long existed.
In the era of oral communication, the “Homer’s Epic” (“Iliad” and “Odyssey”) collected and organized by the ancient Greek poet Homer is a typical example. Chinese interprets “editing” as “collecting materials and organizing them into books”. Going to its shape and taking its meaning, the fundamental feature of this information distribution is that the information is distributed and distributed to the recipient, with the meaning of processing and checking.
No matter social pointsHair, or editorial distribution, they have a long history. Only the specific media that carry these distribution methods are constantly being updated and changed, which brings new possibilities to these distribution methods.
For example, the Internet has achieved cross-regional social connections to some extent through the limitations of social relationships (geography, blood, etc.), and has also shifted the scope of social distribution from the family and offline communities to a wider range. Interest groups.
In the era of the Internet, scientists and engineers are working hard to solve the distribution problem in the information overload environment. The first two representative solutions are the catalogue and search engine – the former, the well-known websites are classified by manual editing. Let users find websites according to categories, such as Yahoo, Hao123, etc.; the latter, let users find the required information by searching for keywords, and solve the limited coverage problem of the catalogue, such as Google, Baidu, etc.
In fact, the ideas of these two solutions are not new, and can largely correspond to the index of the library’s classified collections and encyclopedias.
Overview of the entire history, we can easily find that the information environment is changing, the solution is concrete, but the needs and methods of information distribution are the same. They are all answering a question – how to connect people and information effectively.
Recommended Algorithm: Familiar New Friend
The emergence and universal application of algorithmic distribution means that human beings have begun to use machines to solve information distribution problems on a large scale. The power of human social information distribution has shifted from manpower to partial automation—from “people looking for information” to “information search people”.
In the long river of human social information distribution, although algorithm distribution is a new thing, its mission and foundation are familiar. Thinking from this incision, it is not difficult to answer why the recommended algorithm was born in this era:
First, the new information environment and the power of human information demand call for a new information distribution solution.
In the face of information overload environment and fragmented information consumption scenarios, how to find the information of interest from a large amount of information is very difficult. A search engine that is an important tool can partially meet people’s needs, but it is best suited for scenarios with clear requirements. If users can’t accurately describe their information search needs, or even do not fully understand their needs?
This means that we need a solution that actively proactively distributes information based on our interests and needs. Published as early as 1995In “Being Digital”, Nikola Negroponte puts forward “The Daily Me” and believes that online news will enable the audience to actively choose what they are interested in and predict future information. Personalization.
At the time, this idea might be considered a “daydream.” Because there are natural differences between individuals, and for the overall efficiency of society, people always try to find the “conventional number” of information.
With the development of technology, the emergence of recommendation systems has brought about a possibility for human information distribution: people do not need to provide clear requirements every time, but actively recommend by modeling the information needs of different individuals. Information that meets their interests and needs.
Second, the development of information technology has provided material conditions for the emergence of personalized recommendation systems.
On the one hand, mobile Internet development, everyone is a terminal, which enables the distribution of information to be able to locate different individual users at low cost.
On the other hand, the maturity of AI technology and the evolution of hardware resources provide a technical implementation path for personalized recommendations: the application of machine learning models, the rapid development of deep learning, etc., providing powerful algorithmic tools; The emergence of a distributed machine learning framework, the GPU’s ability to accelerate deep learning, and the emergence of dedicated deep learning chips (TPU, Cambrian) provide another layer of protection.
In 1994, the GroupLens Research Group of the University of Minnesota introduced the first automated recommendation system, GroupLens(1), which proposed an important technology for collaborative filtering as a recommendation system and one of the earliest automated collaborative filtering recommendation systems.
In 1998, Amazon.com launched an item-based collaborative filtering algorithm that pushed the recommendation system to serve millions of users and process millions of items, and produced good quality recommendations.
In October 2006, North American online video service provider Netflix began the prestigious Netflix Prize recommendation system competition. Participants can increase their forecast accuracy by 10% and receive a $1 million bonus. The participating researchers proposed a number of recommendation algorithms, which greatly improved the accuracy of the recommendation and greatly promoted the development of the recommendation system.
In 2016, YouTube published a paper (2) that will enable the deep neural network application recommendation system to find the most likely recommendation results from large-scale optional recommendations.
Since the first recommendation system was born, it has been more than 20 years. Now, the ideas and applications recommended by the algorithm have been deeply embedded in many Internet applications.
For example, personalized reading of content distribution platforms (today headlines, vibrato, etc.), search engine rankings (Google, Baidu, etc.), e-commerce personalized recommendations (Amazon, Taobao, etc., content recommendations for audio and video sites (such as Netflix, YouTube, etc.), social networking sites (Facebook, Weibo, Douban, etc.), and so on.
According to the “2016 China Mobile Information and Information Distribution Market Research Special Report” released by the third-party monitoring organization “Easy View”: In 2016, in the information and information distribution market, the content of algorithm push will exceed 50%. By this year, this proportion must be even bigger.
Today, people are discussing the value of algorithmic distribution. The most frequently mentioned is the improvement of the efficiency of information distribution. It is manifested in: liberating part of the manpower, and breaking through the restrictions imposed by manpower on information distribution, realizing the content of long tail. Effective distribution to more efficiently match people and information.
However, there is still a layer of meaning that is less touched: personalized recommendations through algorithms that truly focus on and understand individuals. Each individual is a “terminal” with a different meaning, rather than always placing the individual in the group for overall understanding. That is, Negroponte said, “In the case of digital survival, I am ‘I’, no longer a ‘subset’ in demographics.” – This is also “personal” (personalization) The meaning of “person” is in it.
The algorithm has more possibilities in front of humanity
The algorithm intelligently matches information, but it is based on people.
Even if the recommendation algorithm develops more maturely, people will inevitably have some confusion in their daily interaction with the algorithm: sometimes, I hope that the algorithm is “smart” and more understanding of myself; sometimes, I don’t want to pay attention to myself. The content, but also want to see public hotspots; sometimes, will guess that in addition to these needs, will there be other potential interests? …
Today, the criticism of content recommendation includes narrowing the horizons, vulgarizing information, and marginalizing people. These sounds fundamentally reflect the eternal concerns of human beings: the breadth and height of information, and Human subjectivity. Faced with these questions, perhaps turning to a holistic and developmental perspective is more conducive to us understanding the problem.
First of all, algorithm recommendation is important, but it is not all. Humans have a variety of information needs scenarios, and different information distribution methods and tools work together to meet the needs of users. The specific tools of these distribution methods may be different at different stages, but they do not completely replace each other in essence.
A simple example: suppose a primary movie hobbyIf you want to watch a movie on the weekend, is there a chance? If he wants to see Kubrick’s work today, he may open the search box directly and search for the director of “Kubrick” to see what his director’s work has not seen before; if he does not have a specific idea, It is possible to open a personalized recommendation app. In the information flow that is familiar with your preferences, brush and see if there is any movie of interest; of course, if he is lucky, WeChat adds a movie enthusiast, you can also directly recommend the other party. unit.
From this example, you can see that the search engine satisfies the active lookup needs of the user with a clear purpose; the recommendation system can help them discover new content of interest when the user has no clear purpose – from this In the sense, “recommendation” and “search” are actually two complementary tools that meet people’s different needs.
When the personalized recommendation application develops rapidly, people may involuntarily assume that it occupies all of their information scenes; however, in reality, a person’s access to information in daily life is far more than we think. To be more enriched – an experiment conducted by scholars such as Seth Flaxman in 2016 also proved this conclusion (3).
The study asked 50,000 participants to report on their own sources of news media recently, and to directly monitor and record their actual news consumption behaviors, including web browsing history, through electronic means. After comparing the two data, the study finally found that people’s actual media consumption is more diverse than they think.
Further, fundamentally, the algorithm is the idea of using intelligence to solve the problem of information distribution, rather than an absolute and stereotyped operation, and it is constantly evolving. Algorithms are not opposite to editing and socializing. Combining the three can help achieve more effective information matching.
In the book Content Algorithm, the author compares the algorithm to “a basket, everything can be loaded inside”: the algorithm is based on the abstraction and modeling of our understanding of the real world, all the factors we care about ( Edit distribution, social distribution) can be converted into reference factors recommended by the algorithm.
The recommended system for practical applications usually uses a variety of recommendation algorithms to improve the personalization, diversity, and robustness (ie, robustness) of the recommendation system. For example, the content-based recommendation algorithm is used to solve the cold start problem of users and content; after having certain user behavior data, user-based collaborative filtering (UserCF) and item-based collaborative filtering are comprehensively used according to the needs of the business scenario ( ItemCF), matrix decomposition or other recommended algorithms for offline calculation and model training, and comprehensively consider the user’s social network data, time correlation and geographic data for recommendation.
At the same time, manual editing also plays a key role. For example, in today’s headline platform, the content is checked by manual review and machine algorithms. One has a good recommendation mechanism and rulesThe platform can promote the dissemination of high-quality content, thus promoting the development of the content ecology. In the new technology environment, the value of professional content production and editing teams will not only fade, but will become more and more important.
Finally, from the discussion of people around the distribution of algorithms, we can see the two pairs of eternal needs of people facing information – personal and public, known and unknown. Humans always hope that the two can achieve a dynamic balance, and this balance point often varies from person to person. This provides motivation and development for the development and improvement of the algorithm.
For individuals, an information ecology that tends to be ideal may need to be social, group, and individual, taking into account the height and breadth of information—some problems, algorithms can solve, and are trying to solve; There are some problems, and humans themselves may not be able to solve problems well. In the end, they must continue to return to human nature. The driving force behind the development and perfection of information distribution technology lies in people, and lies in the eternal pursuit of people’s ideal mode of information distribution. In this process, people always have their unique value and initiative, and adhere to the “technology for people.”
End
Algorithm distribution is the future, it is the product of intelligent distribution in the era of information overload; algorithm distribution will eventually become a thing of the past, because the development of the next generation technology can always break through the imagination of the contemporary people, just like the Song Dynasty people can not Imagine the mobile internet. But in any case, the pace of human pursuit of information will not stop. This pursuit is the driving force behind the flow of information.
After questioning the “previous life” and “this life” of the recommendation algorithm, then in the future of technology development, what is the “afterlife” of the algorithm?