Microsoft may have partnered with OpenAI and its GPT large-scale language model to help create generative AI services like Copilot (formerly the Bing chat tool). However, the company is also developing its own language model. This week, Microsoft Research announced the release of Orca2, the second version of its Orca language.
Microsoft said in a blog post that Orca2 is designed for small-scale LM, but can still be used to answer complex questions such as LLM. Orca2 comes in two sizes (7 billion and 13 billion parameters) and is built in part using Llama2LLM, which Microsoft helped Meta launch earlier this year. The company fine-tuned the Llama2-based model "based on tailor-made, high-quality synthetic data."
Microsoft says this makes the Orca2 model comparable to other language models "5-10 times larger" in solving problems:
Orca2 is trained using an extended, highly customized synthetic dataset. The generation of these training data teaches Orca2 various reasoning techniques, such as step-by-step processing, recall and then generation, recall-inference-generation, extraction-generation and direct answer, and also teaches it to choose different solution strategies for different tasks.
The Orca2 model has conducted a series of benchmark tests with large language models such as Llama2 and WizardLM. The test content includes "language understanding, common sense reasoning, multi-step reasoning, mathematical problem solving, reading comprehension", etc.
Microsoft’s official blog stated: Our preliminary test results show that Orca2’s performance significantly exceeds that of similar-sized models. It also achieved performance levels similar to or better than those of models at least 10 times larger, demonstrating the potential of equipping smaller models with better inference capabilities.
While Microsoft acknowledges that Orca2 does have limitations, testing so far shows "the potential for future advancements." Microsoft will release Orca2 as an open source project so that others can develop it.