801: Merged LLMs Are Smaller And More Capable — with Arcee AI's Mark McQuade and Charles Goddard

Evolutionary Merging

Evolutionary model merging is revolutionizing how we combine multiple large language models (LLMs) into a single, efficient entity. Mark McQuade explains that this technique uses evolutionary algorithms to optimize model parameters, allowing the merging of capabilities without increasing model size 1. This approach has been successfully applied in real-world scenarios, such as with Thomson Reuters, where a 7 billion parameter model outperformed existing closed-source models, offering significant cost savings 2.

The evolutionary approach allows us to take multiple LLMs and merge them into a single model that performs well at the desired objective.

--- Jon Krohn

This method, while requiring GPUs, provides a powerful tool for efficient model merging, leveraging expertise coded into tools like MergeKit 3.

   

Efficient Strategies

Efficient merging strategies are crucial for maximizing the potential of LLMs without the need for extensive hardware. Mark McQuade highlights MergeKit's ability to merge models on limited hardware, such as a ten-year-old laptop, making it accessible for those without high-end GPUs 4. This flexibility allows for significant cost reductions and faster deployment of models in production environments 5.

MergeKit enables the merging of large models on minimal hardware, offering a cost-effective solution for model deployment.

--- Jon Krohn

By combining pre-trained weights into a single network, MergeKit captures the strengths of multiple models, providing a powerful tool for enterprises to efficiently utilize their data 6.

   

Sparse Upcycling

The sparse upcycling technique offers a novel approach to enhancing model efficiency by focusing on specific model weights. Mark McQuade describes how this method allows for the creation of sparse models that activate only a portion of parameters, significantly reducing computational costs 7. This technique is particularly beneficial for large-scale models, providing up to 75% savings on cost and inference time 8.

Sparse upcycling allows for fine-tuning models to be experts in different tasks, optimizing performance while minimizing resource use.

--- Jon Krohn

While primarily reserved for those with extensive computational resources, sparse upcycling presents exciting possibilities for future advancements in model efficiency 9.