•
MLC Community
TL;DR
Machine Learning Compilation (MLC) makes it possible to compile and deploy large-scale language models running on multi-GPU systems with support for NVIDIA and AMD GPUs with high performance. Specifically, we run 4-bit quantized Llama2-70B at 34.5 tok/sec on two NVIDIA RTX 4090 and 29.9 tok/sec on two AMD Radeon 7900XTX. The same solution also scales well beyond two GPUs. Our evaluation shows the performance consistently improves up to eight A10G GPUs when running Llama2-70B and CodeLlama 34B.
Background
Demand for access to powerful GPUs exceeds supply, leading to their increasing cost and limited availability. A significant driver behind this demand is that the appetite for large-scale Large Language Models (LLMs) is strong within academia and industry. However, many of the largest models, such as Meta’s Llama2-70B, face a bottleneck due to their size and cannot be accommodated on a single less-powerful GPU. To address this challenge and assist the GPU-poor during this scarcity era, an ideal LLM serving framework should be universal and capable of deploying on a broader spectrum of devices, highly performant, and able to scale up with the addition of more GPUs.
Machine learning compilation (MLC) techniques offer such a solution, which allow universal deployment on all viable GPUs. Backed by TVM Unity, the latest generation of Apache TVM, MLC LLM is capable of generating highly performant code across all GPU backends, such as CUDA, ROCm, Vulkan, OpenCL, Metal, and WebGPU, achieving state-of-the-art performance across NVIDIA, AMD, and Intel GPUs. Most importantly, unlike a traditional C++ compiler, it compiles for both single-node and multi-GPU and distributed use cases, as machine learning necessitates.
In this blog post, MLC LLM demonstrates multiple advantages in serving large-scale LLMs:
- Performance: It compiles and generates highly performant code for multi-GPU inference.
- Universal deployment: It enables universal deployment of LLMs onto GPUs across a variety of vendors and architectures.
- Scalability: It scales efficiently and provides more acceleration as the number of GPUs increases.
MLC-Powered Multi-GPU Inference
This section demonstrates MLC LLM’s multi-GPU capability by answering the following questions via experiments:
- Performance: How fast is the solution? Is MLC LLM sufficiently performant to serve large-scale LLMs?
- Scalability: How well can we scale? Will the latency meaningfully improve with more GPUs added to the system?
- Universal deployment: Which GPU backends does MLC LLM support out of the box?
Settings
We focus on auto-regressive decoding performance with batch one latency in this blog post because minimum latency is often an important consideration for application builders and a core aspect of good inference performance (prefill = 8, decoding = 256). Long-context prefilling and continuous batching performance are left for future blog posts.
GPUs. We have chosen GPUs from two representative groups: server- and consumer-class GPUs. In the server class, we use A100 (80GB) and A10G (24GB), and in consumer-class GPUs, we use NVIDIA’s RTX 4090 (24GB) and AMD Radeon 7900 XTX (24GB). All numbers are based on PCIe, not NVLink.
Models. We focus on sufficiently large models; in our case, CodeLlama-34B and Llama2-70B. The 4-bit quantized model could fit into 2 GPUs in all our experimental settings, while FP16 numbers are provided whenever VRAM is sufficient.
Drivers and OS. All the experiments are run on Ubuntu 22.04. For NVIDIA GPUs, we use CUDA 12.1, except for vLLM, which only supports CUDA 11.8 at
