domeet webmaster

Because of the fog of war, the two players in Dota 2 can only see the local situation in the whole game, and the other parts of the information are hidden.

Compared to AlphaGo, the AI ​​system for Dota2 has a batch size that is 50-150 times larger, the model is 20 times larger, and the training time is 25 times longer.

Key Point 2: How AI Learns to Play Dota2

Humans play Dota2 to make decisions in real time with keyboard, mouse, etc. As mentioned earlier, OpenAI Five makes an action every 4 frames, which is called a timestep. During each timestep, OpenAI will receive data such as blood volume and location.

The same information is received by humans and OpenAI Five in completely different ways.

When an artificial intelligence system issues an action instruction, it can be thought of like this.

Behind AI is a set of neural networks. policy (π) is defined as a function from observation data to action probability distribution, which is an RNN neural network with 159 million parameters. This network is mainly composed of a single-layer, 4096-unit LSTM.

The structure is shown below:

LSTM contributes 84% ​​of the parameters in this model.

The players ’training uses the extended version of the near-end strategy optimization (PPO) method, which is also the default reinforcement learning of OpenAI now Training methods. The goal of these agents is to maximize the exponential decay and sum of future rewards.

In the process of training strategy, OpenAI Five does not use human game data, but through self-game. On the issues of Go and Chess, similar training is also applied.

Among them, 80% of the opponents are clones with the latest parameters, and 20% of the opponents are clones with the old parameters. After every 10 iterations, the newly trained clone is marked as the old predecessor. If the currently training AI defeats a rookie or an old predecessor, the system will update the parameters based on the learning rate.

According to the OpenAI CTO’s previous statement, before defeating OG, OpenAI Five has practiced the equivalent of 45,000 Dota. The daily training volume of AI is equivalent to 180 years of human games.

Point Three: Calculations and Hyperparameters

Training such a complex AI system will definitely consume a lot of resources.

OpenAI estimated the GPU consumption for optimization. The final conclusion is that OpenAI Five’s GPU computing usage is around 770 ± 50 ~ 820 ± 50 PFlops / s · days. Today, the new and stronger Rerun, in the following two months of training, the GPU computing consumption is about 150 ± 5 PFlops / s · days.

Again, OpenAI announcedIs just the amount of calculation used for optimization, only a small part of all the overhead in training, which accounts for about 30%.

Before, OpenAI also revealed that the daily training of OpenAI Five requires 256 P100 GPUs and 128,000 CPU cores.

As for the hyperparameters of the entire neural network, in the paper, OpenAI said that when training Rerun, it has further simplified the hyperparameters based on experience. In the end, they changed only four key hyperparameters:

• Learning Rate • Entropy penalty coefficient • Team Spirit • GAE time horizon

Of course, OpenAI also said that there is room for further optimization of these hyperparameters.

Key point four: Not all self-study

Finally, there is one more point to emphasize.

OpenAI clearly stated in the thesis that in the process of learning Dota2, the AI ​​system does not rely entirely on reinforcement learning and self-learning, and the enlightenment also uses some human knowledge. This is different from later AlphaGo Zero.

Some game mechanics are scripted programs. For example, the order in which heroes buy equipment and learn skills, the control of messengers, and so on. OpenAI stated in the paper that there are some historical reasons for using these scripts, as well as cost and time considerations. However, the thesis also states that these can also be completed through self-study.