NVIDIA teamed up with research teams from GeorgiaTech, UMD and HKPU to launch a new visual language model - NVEagle.It is reported that NVEagle can understand complex real-life scenes and provide better interpretation and response through visual input.
The core of its design is to convert images into visual markups and then combine them with text embeddings to improve the understanding of visual information. NVEagle includes three versions: Eagle-X5-7B, Eagle-X5-13B and Eagle-X5-13B-Chat. Among them, the 7B and 13B versions are mainly used for general visual language tasks, while the 13B-Chat version is specifically fine-tuned for conversational AI and can better interact based on visual input.
A highlight of NVEagle is the adoption of a Mixed Expert (MoE) mechanism, which can dynamically select the most appropriate visual encoder based on different tasks, which greatly improves the processing capabilities of complex visual information.The model has been published on HuggingFace for the convenience of researchers and developers.