Google just released Gemini, its most powerful suite of artificial intelligence models yet, but the company has been accused of lying about its performance. A Bloomberg column claimed that Google misrepresented Gemini's performance in a recent video. Columnist Parmy Olson said that Google played an impressive "What the AIquack" hands-on video at its launch event earlier this week. In the video, Gemini looked very capable - maybe too powerful.
This six-minute video demonstrates Gemini’s multi-modal capabilities (e.g., spoken dialogue prompts combined with image recognition). Gemini appears to be able to quickly recognize images (even connected pictures), react within seconds, and track paper balls in a cup and ball game in real time. Of course, humans can do all of this, but this is an AI that can recognize and predict what will happen next.
Click on the video description on YouTube, though, and Google has an important disclaimer: "For demonstration purposes, latency has been reduced and Gemini's output has been shortened for simplicity."
This is where Olson is dissatisfied. According to her Bloomberg article, Google admitted when asked for comment that the video demonstration was not live with voice prompts, but instead used still image frames from the original footage and then wrote out text prompts for Gemini to respond to. Olson wrote: "This is very different from what Google seems to be implying: that one can have a smooth voice conversation with Gemini, while Gemini observes and responds to the world around it in real time. To be fair, Google often edits demo videos, especially since many companies want to avoid any technical glitches that come with live demonstrations. A slight adjustment is often Yes. But Google has had problems with video demos. People were skeptical about Google's Duplex demo (remember Duplex? Duplex was an AI voice assistant that could call hair salons and restaurants to make reservations). And pre-recorded videos of AI models tend to make people even more skeptical."
In this case, Olson believes that Google is "showing off" to mislead people into not knowing that Gemini is still lagging behind OpenAI's GPT.
Google doesn't agree with this. When asked about the authenticity of the demo, Google pointed to an article by Oriol Vinyals, vice president of research and head of deep learning at Google DeepMind (who also co-leads Gemini), explaining how the team created the video.
Vinyals said: "All user prompts and output in the video are real and have been shortened for brevity. The video showcases a multi-modal user experience built with Gemini. We made it to inspire developers."
He added that the team gave Gemini pictures and text and asked it to predict what would happen next.
That's certainly one way to handle the situation, but it might not be the right fit for Google -- which, at least in public eyes, has been taken aback by OpenAI's runaway success this year. If Google wants to incentivize developers, it can't do so through carefully edited promotional videos that could misrepresent AI's capabilities. But by letting journalists and developers actually experience the product. Let people do stupid things with Gemini in a small public beta. Let's see how powerful it is.