Who can make the first breakthrough in the popularity of autonomous driving?
Editor’s note: This article is from WeChat public account “Silicon Valley Insight” (ID : Guigudiyixian), author Chelsea Yang.
At the end of 2019, the self-driving giant Waymo announced the acquisition of British company Latent Logic for the improvement of simulation technology. Latent Logic is a British self-driving car company separated from the Department of Computer Science at the University of Oxford. The AI technology developed by it is to imitate learning to build realistic simulations of human driving behavior, thereby developing scalable and Efficient autonomous driving test.
In fact, Waymo has been working on the development of simulation technology, which is also one of its pillar projects for autonomous driving. And the acquisition of Latent Logic can help Waymo achieve more realistic simulation technology. Latent has a machine learning algorithm called imitation learning, which can simulate human behavior, such as driving motorists, motorcyclists, and passersby. Model human imperfect driving behavior and driving errors. The policy technology can simulate these behaviors more realistically, and then better help Waymo make predictions and plans for autonomous driving.
Almost at the same time, Uber is reported to be acquiring simulation software development company Foresight. Foresight was founded in Silicon Valley in 2017 and provides software and data services for emerging mobile robots such as smart taxis, delivery trucks and flying cars. If it acquires Foresight, Uber can also quickly improve its strength in the simulation test of autonomous driving.
(Uber road test accident vehicle, picture from the Internet)
It seems that the major autonomous driving companies have successively made efforts in simulation technology. So what exactly is simulation technology and why is it so important for the development of autonomous driving?
(Waymo simulation image, picture from the network)
What is simulation technology?
As we all know, the perception of self-driving cars is very important. It is necessary to sensitively recognize objects on the road, such as other vehicles, pedestrians, road signs, traffic lights, buildings, trees, and so on. However, When the weather is bad, the following heavy rain, blizzard, extremely strong sunlight, and nights without lights, the perception and recognition ability will be greatly reduced. For example, when strong light is shining on an object that is easy to reflect light, the camera installed on the self-driving car may be affected by the strong light and incorrectly identify the object.
These extreme conditions do not happen often, but in order to deal with these situations, we need to let a large number of self-driving cars be placed in these scenarios, and then continuously learn related processing capabilities through neural networks and machine learning algorithms. So at this time, the simulation environment provides a perfect virtual reality drive test environment, which can simulate extreme scenarios, so that autonomous vehicles can continuously learn processing skills.
So what are the specific contents of the simulation environment simulation?
-
Simulation of driving scenarios
Autonomous vehicles can be understood as a control loop in essence, which contains four basic elements, the physical environment, sensors, controllers and brakes. When the car is driving in the real environment, the sensor will sense and recognize the objects around the vehicle (such as pedestrians and other vehicles by Liar and Radar). Based on these sensed objects, the controller will control the car to respond accordingly. (Such as a pedestrian, the car will be ordered to stop), and finally the brake will respond to the corresponding drive of the car according to the command issued by the controller (if the pedestrian is signaled to stop, the car will be controlled to let the car Parking)
Driving scenario simulation refers to Autonomous vehicles with complete control loops Autonomous vehicles are placed on virtual reality driving including roads, pedestrians walking at random, road signs, buildings, etc. Environment. By building tens of thousands of driving environments, simulation can test whether the control loop of the car works as expected.
-
Sensor performance simulation
The sensors of self-driving cars mainly include Radar and Lidar. They can detect objects through electromagnetic waves and can be complementary to the camera. When the sunlight is too dazzling to affect the camera shooting, the stability of Radar and Lidar can come in handy.
Simulation and analysis of sensor performance is the most important part of an automated driving simulation system. It mainly uses high-fidelity physical analysis to predict changes in sensor performance, such as radar pattern and gain. In addition, the sensor simulation can predict the performance changes of sensors such as radar in real conditions under extreme weather. Compared with the actual changes of the radar on the spot, the simulation prediction cost is much lower and the time consumption is smaller.
(Waymo sensor system, Picture from the network)
In addition, automated functional safety analysis, electronic hardware, semiconductor, and software algorithms can be simulated using simulation technology. The main models of simulation include sensor models, vehicle dynamics models, pedestrian models, static virtual scene models, dynamic traffic flow models, weather and climate models, driving behavior models, and so on.
-
Waymo simulation practice
Waymo previously submitted a 43-page safety report to the U.S. Department of Transportation. In the third part of the report, the test and verification methods also describe how Waymo’s simulation technology can help it achieve advanced real-world driving skills. . Every day, about 25,000 virtual Waymo self-driving cars are tested in simulation software, and their mileage is as high as 8 million miles.
Waymo’s simulation technology is very mature, and there are many types of simulation. Here is a small example for everyone.See how Waymo simulates a vehicle turning left at an intersection.
There is a crossroad on the street of Arizona, the main battlefield of Waymo Road Test. The self-driving vehicle wants to turn left. At this time, the circular green light in front of it lights up. For human drivers, in this traffic light You need to be very careful and cautious in front of you. You need to determine whether there are many oncoming cars, whether they are moving slowly, and whether you can find gaps in the traffic flow, or whether there are few oncoming cars that are moving quickly. You need to stop and wait for the oncoming car to pass before turning left. If you turn left too early, you may collide with the oncoming car. If you turn left too late, it may cause traffic jams in the rear.
In this case, the simulation simulator can provide a lot of practice opportunities for self-driving vehicles, let the car learn repeatedly in each case, and realize the skills of turning.
The specific implementation process is as follows:
First of all, Waymo needs to be continuously tested, with thousands of tests. Simulate different driving scene tests at the same intersection, and practice repeatedly at different intersections until this newly learned turning skill, so as to improve the speed of universal operation.
Next, create a number of variables to change the driving scene. This process of changing the driving scene was named fuzzing by Waymo. You can change the speed of traffic flow, the time of traffic lights, and increase the number of pedestrians and bike lanes waiting to cross the road to make the driving scene more complicated and close to reality. Then observe how these changing factors affect Waymo’s car driving of.
(Fuzzing model established, pictures from the network)
Finally, detection verification and continuous iteration. When the self-driving car learns to turn left in the simulation environment, this skill will be recorded in the knowledge base and then shared with other vehicles. The skills learned in the simulation will be fed back to the real scene to verify, and then the whole process is improved to ensure that in the end, millions of miles of road tests on real roads can be ensured. Security.
In order to achieve fully integrated autonomous vehicle detection, Waymo spent a lot of money in California to build a completely enclosed test site called Castle, covering 91 hectares, which is not accessible to non-related personnel.
(Castle props warehouse, pictures from the Internet)
This place is more like a closed small town. There are all kinds of road scenes in it, such as highways, railway crossings, and many plastic props, such as pedestrians and roadblocks. In order to help simulation modeling and verification, Waymo set up more than 20,000 traffic simulation scenes in Castle. For example, a skateboarder lay on a skateboard, and a car parked on the side of the road suddenly came down, regardless of the car behind the car changing lanes.
The importance of simulation technology
At present, there are mainly three types of simulation software for autonomous driving on the market. The first category is specialized simulation software for autonomous driving. The second category is simulation software based on game engines. All the engineers who do simulation have a background in games. The third type is a simulator based on robot simulation software.
The simulation software can focus on various extreme and unexpected situations. By fine-tuning and modifying parameters, it can reorganize a variety of new emergent scenarios. In actual road test scenarios, it is difficult to encounter so many unexpected situations, and many extreme situations are very dangerous, such as a child suddenly rushing out of the road, and then on an icy highway, these roads When the situation under test occurs, there is a great risk involved. With simulation technology, these extreme situations can be easily reproduced, saving time and money.
Another advantage of simulation is that in the simulation environment, self-driving cars can achieve large-scale road tests in a short time. Nvidia’s CEO Huang Renxun said at the European Nvidia GPU Developers Conference that using Nvidia’s DGX and Tensor RT 3 for driving simulation, engineers can complete a road test of 480,000 kilometers in 5 hours. At this speed, all the US The road takes only 2 days.
So, Many autonomous driving companies are investing heavily, or developing their own simulation technology, or acquiring some companies specializing in simulation technology. At present, there are many companies specializing in simulation technology like bamboo shoots Rising from the ground, such as the Google-based company Drive.ai, which was established in 2015, has now received $ 77 million in four rounds of financing.
![Uber, Waymo have acquired simulation technology companies, why simulation is so important to autonomous driving
</p>
</div>
<p class=]()
Simulation of driving scenarios
Autonomous vehicles can be understood as a control loop in essence, which contains four basic elements, the physical environment, sensors, controllers and brakes. When the car is driving in the real environment, the sensor will sense and recognize the objects around the vehicle (such as pedestrians and other vehicles by Liar and Radar). Based on these sensed objects, the controller will control the car to respond accordingly. (Such as a pedestrian, the car will be ordered to stop), and finally the brake will respond to the corresponding drive of the car according to the command issued by the controller (if the pedestrian is signaled to stop, the car will be controlled to let the car Parking)
Driving scenario simulation refers to Autonomous vehicles with complete control loops Autonomous vehicles are placed on virtual reality driving including roads, pedestrians walking at random, road signs, buildings, etc. Environment. By building tens of thousands of driving environments, simulation can test whether the control loop of the car works as expected.
-
Sensor performance simulation
The sensors of self-driving cars mainly include Radar and Lidar. They can detect objects through electromagnetic waves and can be complementary to the camera. When the sunlight is too dazzling to affect the camera shooting, the stability of Radar and Lidar can come in handy.
Simulation and analysis of sensor performance is the most important part of an automated driving simulation system. It mainly uses high-fidelity physical analysis to predict changes in sensor performance, such as radar pattern and gain. In addition, the sensor simulation can predict the performance changes of sensors such as radar in real conditions under extreme weather. Compared with the actual changes of the radar on the spot, the simulation prediction cost is much lower and the time consumption is smaller.
(Waymo sensor system, Picture from the network)
In addition, automated functional safety analysis, electronic hardware, semiconductor, and software algorithms can be simulated using simulation technology. The main models of simulation include sensor models, vehicle dynamics models, pedestrian models, static virtual scene models, dynamic traffic flow models, weather and climate models, driving behavior models, and so on.
-
Waymo simulation practice
Waymo previously submitted a 43-page safety report to the U.S. Department of Transportation. In the third part of the report, the test and verification methods also describe how Waymo’s simulation technology can help it achieve advanced real-world driving skills. . Every day, about 25,000 virtual Waymo self-driving cars are tested in simulation software, and their mileage is as high as 8 million miles.
Waymo’s simulation technology is very mature, and there are many types of simulation. Here is a small example for everyone.See how Waymo simulates a vehicle turning left at an intersection.
There is a crossroad on the street of Arizona, the main battlefield of Waymo Road Test. The self-driving vehicle wants to turn left. At this time, the circular green light in front of it lights up. For human drivers, in this traffic light You need to be very careful and cautious in front of you. You need to determine whether there are many oncoming cars, whether they are moving slowly, and whether you can find gaps in the traffic flow, or whether there are few oncoming cars that are moving quickly. You need to stop and wait for the oncoming car to pass before turning left. If you turn left too early, you may collide with the oncoming car. If you turn left too late, it may cause traffic jams in the rear.
In this case, the simulation simulator can provide a lot of practice opportunities for self-driving vehicles, let the car learn repeatedly in each case, and realize the skills of turning.
The specific implementation process is as follows:
First of all, Waymo needs to be continuously tested, with thousands of tests. Simulate different driving scene tests at the same intersection, and practice repeatedly at different intersections until this newly learned turning skill, so as to improve the speed of universal operation.
Next, create a number of variables to change the driving scene. This process of changing the driving scene was named fuzzing by Waymo. You can change the speed of traffic flow, the time of traffic lights, and increase the number of pedestrians and bike lanes waiting to cross the road to make the driving scene more complicated and close to reality. Then observe how these changing factors affect Waymo’s car driving of.
(Fuzzing model established, pictures from the network)
Finally, detection verification and continuous iteration. When the self-driving car learns to turn left in the simulation environment, this skill will be recorded in the knowledge base and then shared with other vehicles. The skills learned in the simulation will be fed back to the real scene to verify, and then the whole process is improved to ensure that in the end, millions of miles of road tests on real roads can be ensured. Security.
In order to achieve fully integrated autonomous vehicle detection, Waymo spent a lot of money in California to build a completely enclosed test site called Castle, covering 91 hectares, which is not accessible to non-related personnel.
(Castle props warehouse, pictures from the Internet)
This place is more like a closed small town. There are all kinds of road scenes in it, such as highways, railway crossings, and many plastic props, such as pedestrians and roadblocks. In order to help simulation modeling and verification, Waymo set up more than 20,000 traffic simulation scenes in Castle. For example, a skateboarder lay on a skateboard, and a car parked on the side of the road suddenly came down, regardless of the car behind the car changing lanes.
The importance of simulation technology
At present, there are mainly three types of simulation software for autonomous driving on the market. The first category is specialized simulation software for autonomous driving. The second category is simulation software based on game engines. All the engineers who do simulation have a background in games. The third type is a simulator based on robot simulation software.
The simulation software can focus on various extreme and unexpected situations. By fine-tuning and modifying parameters, it can reorganize a variety of new emergent scenarios. In actual road test scenarios, it is difficult to encounter so many unexpected situations, and many extreme situations are very dangerous, such as a child suddenly rushing out of the road, and then on an icy highway, these roads When the situation under test occurs, there is a great risk involved. With simulation technology, these extreme situations can be easily reproduced, saving time and money.
Another advantage of simulation is that in the simulation environment, self-driving cars can achieve large-scale road tests in a short time. Nvidia’s CEO Huang Renxun said at the European Nvidia GPU Developers Conference that using Nvidia’s DGX and Tensor RT 3 for driving simulation, engineers can complete a road test of 480,000 kilometers in 5 hours. At this speed, all the US The road takes only 2 days.
So, Many autonomous driving companies are investing heavily, or developing their own simulation technology, or acquiring some companies specializing in simulation technology. At present, there are many companies specializing in simulation technology like bamboo shoots Rising from the ground, such as the Google-based company Drive.ai, which was established in 2015, has now received $ 77 million in four rounds of financing.
![Uber, Waymo have acquired simulation technology companies, why simulation is so important to autonomous driving
</p>
</div>
<p class=]()
Sensor performance simulation
The sensors of self-driving cars mainly include Radar and Lidar. They can detect objects through electromagnetic waves and can be complementary to the camera. When the sunlight is too dazzling to affect the camera shooting, the stability of Radar and Lidar can come in handy.
Simulation and analysis of sensor performance is the most important part of an automated driving simulation system. It mainly uses high-fidelity physical analysis to predict changes in sensor performance, such as radar pattern and gain. In addition, the sensor simulation can predict the performance changes of sensors such as radar in real conditions under extreme weather. Compared with the actual changes of the radar on the spot, the simulation prediction cost is much lower and the time consumption is smaller.
(Waymo sensor system, Picture from the network)
In addition, automated functional safety analysis, electronic hardware, semiconductor, and software algorithms can be simulated using simulation technology. The main models of simulation include sensor models, vehicle dynamics models, pedestrian models, static virtual scene models, dynamic traffic flow models, weather and climate models, driving behavior models, and so on.
-
Waymo simulation practice
Waymo previously submitted a 43-page safety report to the U.S. Department of Transportation. In the third part of the report, the test and verification methods also describe how Waymo’s simulation technology can help it achieve advanced real-world driving skills. . Every day, about 25,000 virtual Waymo self-driving cars are tested in simulation software, and their mileage is as high as 8 million miles.
Waymo’s simulation technology is very mature, and there are many types of simulation. Here is a small example for everyone.See how Waymo simulates a vehicle turning left at an intersection.
There is a crossroad on the street of Arizona, the main battlefield of Waymo Road Test. The self-driving vehicle wants to turn left. At this time, the circular green light in front of it lights up. For human drivers, in this traffic light You need to be very careful and cautious in front of you. You need to determine whether there are many oncoming cars, whether they are moving slowly, and whether you can find gaps in the traffic flow, or whether there are few oncoming cars that are moving quickly. You need to stop and wait for the oncoming car to pass before turning left. If you turn left too early, you may collide with the oncoming car. If you turn left too late, it may cause traffic jams in the rear.
In this case, the simulation simulator can provide a lot of practice opportunities for self-driving vehicles, let the car learn repeatedly in each case, and realize the skills of turning.
The specific implementation process is as follows:
First of all, Waymo needs to be continuously tested, with thousands of tests. Simulate different driving scene tests at the same intersection, and practice repeatedly at different intersections until this newly learned turning skill, so as to improve the speed of universal operation.
Next, create a number of variables to change the driving scene. This process of changing the driving scene was named fuzzing by Waymo. You can change the speed of traffic flow, the time of traffic lights, and increase the number of pedestrians and bike lanes waiting to cross the road to make the driving scene more complicated and close to reality. Then observe how these changing factors affect Waymo’s car driving of.
(Fuzzing model established, pictures from the network)
Finally, detection verification and continuous iteration. When the self-driving car learns to turn left in the simulation environment, this skill will be recorded in the knowledge base and then shared with other vehicles. The skills learned in the simulation will be fed back to the real scene to verify, and then the whole process is improved to ensure that in the end, millions of miles of road tests on real roads can be ensured. Security.
In order to achieve fully integrated autonomous vehicle detection, Waymo spent a lot of money in California to build a completely enclosed test site called Castle, covering 91 hectares, which is not accessible to non-related personnel.
(Castle props warehouse, pictures from the Internet)
This place is more like a closed small town. There are all kinds of road scenes in it, such as highways, railway crossings, and many plastic props, such as pedestrians and roadblocks. In order to help simulation modeling and verification, Waymo set up more than 20,000 traffic simulation scenes in Castle. For example, a skateboarder lay on a skateboard, and a car parked on the side of the road suddenly came down, regardless of the car behind the car changing lanes.
The importance of simulation technology
At present, there are mainly three types of simulation software for autonomous driving on the market. The first category is specialized simulation software for autonomous driving. The second category is simulation software based on game engines. All the engineers who do simulation have a background in games. The third type is a simulator based on robot simulation software.
The simulation software can focus on various extreme and unexpected situations. By fine-tuning and modifying parameters, it can reorganize a variety of new emergent scenarios. In actual road test scenarios, it is difficult to encounter so many unexpected situations, and many extreme situations are very dangerous, such as a child suddenly rushing out of the road, and then on an icy highway, these roads When the situation under test occurs, there is a great risk involved. With simulation technology, these extreme situations can be easily reproduced, saving time and money.
Another advantage of simulation is that in the simulation environment, self-driving cars can achieve large-scale road tests in a short time. Nvidia’s CEO Huang Renxun said at the European Nvidia GPU Developers Conference that using Nvidia’s DGX and Tensor RT 3 for driving simulation, engineers can complete a road test of 480,000 kilometers in 5 hours. At this speed, all the US The road takes only 2 days.
So, Many autonomous driving companies are investing heavily, or developing their own simulation technology, or acquiring some companies specializing in simulation technology. At present, there are many companies specializing in simulation technology like bamboo shoots Rising from the ground, such as the Google-based company Drive.ai, which was established in 2015, has now received $ 77 million in four rounds of financing.
![Uber, Waymo have acquired simulation technology companies, why simulation is so important to autonomous driving
</p>
</div>
<p class=]()
(Waymo sensor system, Picture from the network) Waymo simulation practice
Waymo previously submitted a 43-page safety report to the U.S. Department of Transportation. In the third part of the report, the test and verification methods also describe how Waymo’s simulation technology can help it achieve advanced real-world driving skills. . Every day, about 25,000 virtual Waymo self-driving cars are tested in simulation software, and their mileage is as high as 8 million miles.
Waymo’s simulation technology is very mature, and there are many types of simulation. Here is a small example for everyone.See how Waymo simulates a vehicle turning left at an intersection.
There is a crossroad on the street of Arizona, the main battlefield of Waymo Road Test. The self-driving vehicle wants to turn left. At this time, the circular green light in front of it lights up. For human drivers, in this traffic light You need to be very careful and cautious in front of you. You need to determine whether there are many oncoming cars, whether they are moving slowly, and whether you can find gaps in the traffic flow, or whether there are few oncoming cars that are moving quickly. You need to stop and wait for the oncoming car to pass before turning left. If you turn left too early, you may collide with the oncoming car. If you turn left too late, it may cause traffic jams in the rear.
In this case, the simulation simulator can provide a lot of practice opportunities for self-driving vehicles, let the car learn repeatedly in each case, and realize the skills of turning.
The specific implementation process is as follows:
First of all, Waymo needs to be continuously tested, with thousands of tests. Simulate different driving scene tests at the same intersection, and practice repeatedly at different intersections until this newly learned turning skill, so as to improve the speed of universal operation.
Next, create a number of variables to change the driving scene. This process of changing the driving scene was named fuzzing by Waymo. You can change the speed of traffic flow, the time of traffic lights, and increase the number of pedestrians and bike lanes waiting to cross the road to make the driving scene more complicated and close to reality. Then observe how these changing factors affect Waymo’s car driving of.
(Fuzzing model established, pictures from the network)
Finally, detection verification and continuous iteration. When the self-driving car learns to turn left in the simulation environment, this skill will be recorded in the knowledge base and then shared with other vehicles. The skills learned in the simulation will be fed back to the real scene to verify, and then the whole process is improved to ensure that in the end, millions of miles of road tests on real roads can be ensured. Security.
In order to achieve fully integrated autonomous vehicle detection, Waymo spent a lot of money in California to build a completely enclosed test site called Castle, covering 91 hectares, which is not accessible to non-related personnel.
(Castle props warehouse, pictures from the Internet)
This place is more like a closed small town. There are all kinds of road scenes in it, such as highways, railway crossings, and many plastic props, such as pedestrians and roadblocks. In order to help simulation modeling and verification, Waymo set up more than 20,000 traffic simulation scenes in Castle. For example, a skateboarder lay on a skateboard, and a car parked on the side of the road suddenly came down, regardless of the car behind the car changing lanes.