An embarrassing discovery: the autonomous driving system also discriminates against groups of people. Researchers from King's College London conducted a study and found a loophole after examining more than 8,000 images: the AI-driven pedestrian detection system used by self-driving cars has a detection accuracy of 19.67% lower for children than for adults, and a detection accuracy of dark skin is 7.53% lower than that of light skin. There is not much difference in detection accuracy between genders, with only a 1.1% gap.
This means that children and dark-skinned pedestrians will be harder for self-driving cars to detect than adults and light-skinned pedestrians.
Why is this happening?
Unfriendly to children and dark-skinned people
Let’s look at the experimental process first.
This research team used a data analysis method. They first found 8 pedestrian-specific detection systems that are most commonly used by autonomous driving companies and are also common on the market.
These pedestrian detection systems are then used to collect real scene test data, including actual scenes with different brightness, contrast, weather conditions, etc. These data sets are mainly composed of real street images taken.
They obtained a total of 8311 images in four real scenes, which showed pedestrians in different poses, sizes and occlusion scenarios. The researchers also specially added tags to the pedestrians in the images, with a total of 16,070 gender tags, 20,115 age tags, and 3,513 skin color tags.
The focus of the research is whether the pedestrian detection system of autonomous driving responds the same way when facing different pedestrians, especially whether there will be unfair problems in the three factors of gender, age and skin color.
The detection systems used include ALFNet, CSP, MGAN and PRNet, etc. Among them, ALFNet uses multi-step prediction for asymptotic positioning, which solves the limitations of single-step detection in pedestrian detection.
CSP introduces an anchor-free method by locating the center and scaling pedestrians; MGAN uses visible area bounding box information to guide attention generation and is mainly used for pedestrian detection under occlusion.
After collecting the images, the team used a difference formula to question whether the self-driving system was unfair to the group. MR generally represents the most commonly used performance indicator in pedestrian detection research, MR=1-TP/(TP+FN), where TP (true positive) refers to the number of successfully deleted ground-truth bounding boxes, and FN (false negative) refers to the number of undetected ground-truth bounding boxes.
After calculation, the pedestrian detector’s detection rate for female and male pedestrians is similar, with a difference of 1.1%. However, the difference in age and skin color is larger, reaching 19.67% and 7.52% respectively!
This means that driverless pedestrian detection systems will have more difficulty identifying children and people with darker skin, and these groups will also face greater risks.
What is particularly noteworthy is that these numbers increase to a certain extent at night. Children's EOD (difference between children and adult groups) increases from day to night, the out-of-detection rate increases from 22.05% to 26.63%, and the difference rate of skin color groups (dark and light skin) increases from 7.14% during the day to 9.68% at night.
In addition, compared with men, women have higher delinquency rates in all three factors.
In addition, the research team studied the data under different brightness and contrast conditions. These variables will also have a greater impact on the detection rate.
Among the eight pedestrian detection systems selected, as the brightness decreases, the first-level detection system performs the worst, especially in skin color, where the difference between dark skin and light skin reaches the highest value.
"Fair AI should treat all groups equally, but this does not seem to be the case with driverless cars at present." Dr. Jie Zhang, the author of the study, said.
Why does this happen?
This is mainly because artificial intelligence systems require a large amount of data to train, and once these data are insufficient, it will inevitably be reflected in the performance of artificial intelligence. This also means that the lack of training data has led to certain biases in some artificial intelligence AI.
There are still many unresolved issues
In fact, there is a certain degree of unfairness in artificial intelligence systems, and this is not the first time researchers have studied it.
As early as 2019, research from the Georgia Institute of Technology in the United States showed that people with darker skin are more likely to be hit by self-driving cars on the road than people with fair skin. Researchers analyzed the methods of self-driving cars to detect objects and analyzed a total of 3,500 photos of people with different skin colors.
Finally, it was concluded that driverless technology was 5% less accurate on average when identifying people with dark skin.
Although these studies do not involve driverless cars that are already on the road, they will undoubtedly make people more alert to driverless technology.
A large part of the reason why autonomous driving is difficult to implement is that it cannot truly replace humans in responding to pedestrians and road conditions in a timely manner.
In 2018, a driverless car of ride-hailing service giant Uber hit and killed a person in Tempe, Arizona. This was the first driverless accident. "Lack of time to react" was a major problem.
Some time ago, California voted to allow two major autonomous taxis, Cruise and Waymo, to operate around the clock in San Francisco. This news aroused dissatisfaction among the American public because autonomous taxis often cause accidents.
The car's driverless system can identify road conditions in a variety of ways, such as the lidar mounted on the roof, which can produce three-dimensional images of the car's surroundings many times per second. It mainly uses infrared laser pulses to reflect objects and transmit the signal to the sensor, which can detect stationary and moving objects.
However, when encountering extreme weather, such as dense fog or heavy rain, the accuracy of lidar will be greatly reduced.
Short-range and long-range optical cameras can actually read signals and determine the color of objects and other more detailed objects, which can make up for the shortcomings of lidar.
In order to increase recognition capabilities, many domestic unmanned driving systems have adopted a hybrid perception route, which is realized through lidar and camera vision technology. Visual perception takes precedence over radar perception, with visual perception being the main one and radar perception being the supplement.
But Tesla is a big fan of "pure visual perception," with Musk once saying lidar is like the human appendix. However, this has also led to Tesla being sued several times due to accidents.
In fact, even the hybrid perception route still needs to overcome many challenges.
For example, pedestrians imaged at long distances usually have small targets, which results in low resolution and insufficient positioning accuracy. This is also one of the reasons why children have a high rate of delinquency. Secondly, different postures of pedestrians will also lead to inaccurate detection by the algorithm, and pedestrian detection will be affected by the background, such as the intensity of light, changes in weather, etc., which will affect the judgment.
Finally, there are obstacles. Overlapping targets and occlusions also have a great impact on algorithm recognition.
Chinese scholars lead research
The full title of this paper introducing the fairness of autonomous driving systems is "Dark-skinned people face more risks on the streets: Uncovering fairness issues in autonomous driving systems." The paper was published in the magazine "New Scientist".
The research team of the paper is from King's College London. There are 6 authors listed in the paper, including Xinyue Li, Ying Zhang, and Xuanzhe Liu from Peking University in China, Zhenpeng Chen and Federico Casaro from University of London, UK, and Jie M. Zhang from King's College London.
Jie M. Zhang is currently an Assistant Professor at King's College London. Her research focuses on combining software engineering research with artificial intelligence research to improve the credibility of software. She was a researcher at the University of London and received her PhD in computer science from Peking University in China.
As a Chinese scholar, Jie M. Zhang's achievements in China are also remarkable. She was named "one of the top fifteen young female scholars in China" in March this year. She has also been invited to give keynote speeches on the credibility of machine translation many times. She and her team have also conducted research and analysis on the learning ability of artificial intelligence many times.
Regarding the lack of fairness in pedestrian detection systems, Jie M. Zhang said that automakers and governments need to jointly formulate regulations to ensure the safety and fairness of autonomous driving systems.
In fact, there have been artificial intelligence recruitment software and facial recognition software before, and the accuracy of black women is not as good as that of white men. Now, if self-driving cars have recognition misunderstandings, the consequences may be more serious.
"In the past, ethnic minorities may have been deprived of the convenience they deserve because of some software." JieM. Zhang said that now they may face more serious harm, even personal injury.