Foreign media broke the news that Meta is urgently developing a new open source large model to benchmark GPT-4, and the number of parameters is several times larger than Llama2. Is Meta going to subvert the AI community again?According to the foreign media "Wall Street Journal", Meta is stepping up the development of a new large language model. Its capabilities will be fully aligned with GPT-4 and is expected to be launched next year.
The news also specifically emphasized that Meta’s new large language model will be several times larger than Llama2, and it will most likely be open source and support free commercial use.
Since Meta "accidentally" leaked LlaMA at the beginning of the year, to the open source release of Llama2 in July, Meta has gradually found its unique position in this AI wave - the banner of the AI open source community.
01
The staff is constantly shaken, and the model ability is flawed. We rely on open source to return to the main table.
At the beginning of the year, after OpenAI detonated the technology industry with GPT-4, Google and Microsoft also launched their own AI products.
In May, U.S. regulators invited CEOs of leading companies they considered relevant to the AI industry to hold a roundtable meeting to discuss the development of AI technology.
OpenAI, Google, and Microsoft were all invited, and even the startup Anthropic was included, but Meta was not present. The official response to Meta's absence at the time was: "We only invite the top companies in the AI industry."
Good things didn't happen to Meta, but troubles kept coming.
First, a letter of inquiry from Congress was sent directly to Xiao Zha in early June, asking him in stern terms to explain the causes and consequences of the LlaMA leakage in March.
In the following months, even after the release of Llama2, the AI team that Meta had spent a lot of money to build was still gradually falling apart.
In Llama2's acknowledgments, three of the four mentioned teams that first initiated this research have resigned, and currently only Edouard Grave is still in Meta.
Industry giant He Yuming will also leave Meta and return to academia.
According to a recent breaking article in The Information, Meta’s AI team has been experiencing constant friction due to competition for internal computing power, and personnel have been leaving one after another.
Against this background, Xiao Zha himself should also be very clear that Meta’s own large language model is indeed unable to compete with the industry’s most cutting-edge GPT-4.
Whether it is benchmark testing in various directions or user feedback, the gap between Llama2 and GPT-4 is still relatively large.
In various benchmark tests, there is still a big gap between open source Llama2 and GPT-4.
The actual experience of netizens also constantly emphasizes that GPT-4 is still much ahead of Llama2
Therefore, Xiao Zha decided to let Meta continue to run wildly on the road of model open source.
Perhaps the logic behind Xiaozha is this: Meta model capabilities are average and cannot beat the closed-source big guys, so there is no point in hiding them. Then simply open source and let the AI community continue to iterate based on its own models to expand the influence of its products in the industry.
Moreover, Xiao Zha has said in public more than once that the iteration of his own model by the open source community will inspire his technical team to develop more competitive products in the future.
Xiao Zha emphasized in Fridman's podcast that open source allows Meta to draw inspiration from the community, and that Meta may launch a closed source model in the future. See: https://lexfridman.com/mark-zuckerberg-2/
And facts have proven that Meta’s choice is indeed correct.
Although it is inferior to Google and OpenAI in terms of computing resources and technical strength, open source models such as Meta's Llama2 are still second to none in their appeal to the open source community. As Llama2 slowly becomes the "technical base" of the AI open source community, Meta has also found its own ecological niche in the industry.
The most obvious sign is that in the closed-door meeting of the Congress on AI that will be held in September, Xiao Zha finally became a guest of the regulators. Together with the CEOs of the most cutting-edge companies in the industry such as Google and OpenAI, he served as a representative to express his own voice on the regulation of the AI industry.
And if the new model launched by Meta next year can continue to make progress and gain the same capabilities as GPT-4, on the one hand, it will allow the open source community to continue to close the gap with the closed source giants, confirming the statement that "the gap between the open source community and the most advanced level in the industry is about one year."
On the other hand, Xiao Zha also revealed in the interview that if the capabilities of large models are further improved in the future, Meta may launch its own closed-source model. If the new model can further approach the industry SOTA, it may not be far from Meta launching its own closed-source model.
Although Meta seems to have temporarily lagged behind in this wave of AI, Xiao Zha’s ambition is not willing to be just a follower.
Under the guidance of the "AI Big Three" Yann Lecun, Meta is also preparing to subvert the entire industry.
02
The future of Meta
So, after this mysterious large model that is legendary to be comparable to GPT-4, what will MetaAI look like in the future?
Because there is no specific information yet, we can only make some guesses, such as starting from the attitude of MetaAI chief scientist LeCun.
GPT, the popular fried chicken, has always been the artificial intelligence development route that LeCun criticized and despised.
On February 4 this year, LeCun stated bluntly, “On the road to human-level AI, large-scale language models are completely a crooked path.”
He believes that this kind of large model that generates autoregression based on probability will not survive for at most 5 years, because these artificial intelligences are only trained on a large amount of text, and they cannot understand the real world.
So these models can neither plan nor reason, they only have contextual learning capabilities.
Seriously, these artificial intelligences trained on LLM have almost no "intelligence" at all.
What LeCun is looking forward to is a "world model" that can lead to AGI.
World models can learn how the world works, learn more quickly, plan for completing complex tasks, and respond to unfamiliar new situations at any time.
This is different from LLM, which requires a lot of pre-training. The world model can find patterns from observations, adapt to new environments, and master new skills like humans.
Compared with OpenAI's strategy of continuous improvement and deepening in the field of LLM, Meta strives for diversified model development.
On June 14 this year, Meta released I-JEPA, a "human-like" artificial intelligence model, which is also the first AI model in history based on key parts of LeCun's world model vision.
Paper address: https://arxiv.org/abs/2301.08243
I-JEPA is able to understand abstract representations in images and acquire common sense through supervised learning.
And I-JEPA does not require additional artificial knowledge as an aid.
Later, Meta launched Voicebox, a new breakthrough speech generation system based on a new method proposed by MetaAI - flow matching.
It can synthesize speech in six languages, perform operations such as denoising, editing content, and converting audio styles.
Meta also released general-purpose embodied AIagents.
Through language-guided skill coordination (LSC), the robot can move and pick freely in a partially pre-mapped environment.
Meta is also different in the development of multimodal models.
ImageBind, the first artificial intelligence model capable of binding information from six different modalities.
It gives machines comprehensive understanding, linking objects in photos to their sounds, three-dimensional shapes, temperatures and how they move.
RoboAgent, jointly developed by MetaAI and CMU_Robotics, allows robots to acquire a variety of non-trivial skills and promote them to hundreds of life scenarios.
At the same time, all these scenarios have an order of magnitude less data than previous work in the field.
Regarding the model that was revealed this time, some netizens expressed the hope that they will continue to open source code.
However, some netizens said that Meta will not start training until early 2024.
But what is gratifying is that Meta still released a signal that it will continue to adhere to its original strategy.