A new report from Stanford University's HAI (Human-Centered Artificial Intelligence) finds that none of the well-known developers of the underlying models of artificial intelligence, including companies such as OpenAI and Meta, have released enough information about their potential impact on society.
Today, Stanford HAI released its Underlying Model Transparency Index, which tracks whether the creators of the 10 most popular artificial intelligence models disclose information about their work and how people use their systems. Among the models it tested, Meta’s Llama2 scored highest, followed by BloomZ and then OpenAI’s GPT-4. But it turns out that none of them scored particularly high.
Other models evaluated include Stability’s StableDiffusion, Anthropic’s Claude, Google’s PaLM2, Cohere’s Command, AI21Labs’ Jurassic2, Inflection’s Inflection-1, and Amazon’s Titan.
The researchers acknowledge that transparency can be a fairly broad concept. Their definition is based on 100 indicators that provide information about how the model is built, how the model works, and how people use the model. They parsed the public information about the model and gave each a score, noting whether the companies disclosed partners and third-party developers, whether they told customers whether their models used private information, and a host of other questions.
As the company released its research on model creation, the Meta score was 53%, with the highest score in model basics. Open-source model BloomZ follows closely behind with 50% support, and GPT-4 with 47% support - tied for stable diffusion despite OpenAI's relatively locked-in design approach.
OpenAI refuses to publish most of its research results and does not reveal the source of its data, but GPT-4 manages to rank highly because there is a lot of information available about its partners. OpenAI works with a number of different companies to integrate GPT-4 into their products, resulting in a wealth of public details available for review.
However, Stanford researchers found that none of the model's creators revealed any information about the social impact, including where to file privacy, copyright or bias complaints.
Rishi Bommasani, society director at Stanford University's Center for Fundamental Modeling and one of the index's researchers, said the index's goal is to provide a benchmark for governments and companies. Some proposed regulations, such as the EU's Artificial Intelligence Bill, may soon force developers of large-scale underlying models to provide transparency reporting.
"What we are trying to achieve with the index is to make the model more transparent and break down very unclear concepts into more concrete things that can be measured," Bomasani said. The group focused on one model from each company to make comparisons easier.
Generative AI has a large and active open source community, but some of the largest companies in the field do not share research or their code publicly. Although OpenAI has the word "open" in its name, it no longer distributes its research, citing competition and security concerns.
Bommasani said the organization is open to expanding the scope of the index but in the meantime will stick with the 10 base models it has already evaluated.