Generative AI services can be used to generate generic text snippets, incredible images, and even code scripts in various programming languages. However, when LLM is used to produce questionable or meaningless reports, the results can be detrimental to project development to a great extent.
Daniel Stenberg, the original author and lead developer of the curl software, recently wrote about the problematic impact of LLM and artificial intelligence models on the project. The Swedish programmer noted that the team has a bug bounty program that offers real-money rewards to hackers who find security issues, but that superficial reports created through AI services are becoming a real problem.
Curl's bug bounty program has paid out $70,000 in rewards so far, Stenberg said. The programmer received a total of 415 vulnerability reports, 77 of which were "informational" reports, and 64 were ultimately confirmed as security issues. A significant number of reported issues (66%) were neither security issues nor common vulnerabilities.
Generative AI models are increasingly used (or proposed to be used) as a way to automate complex programming tasks, but LLM is known for its "illusion" and remarkable ability to deliver meaningless results while sounding absolutely confident in its output. In Sternberg’s own words, AI-based reporting looks better and seems to make sense, but “better garbage” is still garbage.
Programmers would have to spend more time and effort on the report before turning it off, Sternberg said. AI-generated garbage doesn’t help the project at all, as it takes developers’ time and energy away from productive work. The curl team needs to properly investigate every report, and artificial intelligence models can exponentially reduce the time required to write error reports that may not even exist.
Sternberg cited two fake reports that were likely created by artificial intelligence. The first report claims to describe a real security vulnerability (CVE-2023-38545) that has not even been disclosed yet but is full of "classic AI-style hallucinations." Sternberg said facts and details from old security issues were jumbled together to create something new that had "no connection" to reality.
Another report recently filed on HackerOne describes a potential buffer overflow vulnerability in WebSocket handling. Sternberg tried to raise some questions about the report, but ultimately concluded that the bug wasn't real and that he was likely talking to an AI model rather than a real person.
The programmer said that artificial intelligence can do "a lot of good things" but can also be used to do the wrong things. In theory, LLM models can be trained to report security issues in a productive way, but we still need to find "good examples" of this. Sternberg said that over time, AI-generated reports will become more common, so teams must learn how to better trigger "AI-generated" signals and quickly dismiss those false reports.