Large Language Models (LLMs)

Projects

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

arxiv 2024

(

)

VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation.

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

arXiv

(

)

By selectively applying full attention to critical attention heads and using "Streaming Attention" on others, DuoAttention significantly reduces both pre-filling and decoding memory usage and latency for long-context LLMs, while maintaining their long-context capabilities.

HART: Efficient Visual Generation with Hybrid Autoregressive Transformer

arXiv 2024

(

)

HART is an autoregressive transformer that generates high resolution images with comparable quality to diffusion models, while offering 4.5-7.7x higher throughput.

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

ArXiv 2024

(

)

We introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache, and implement QServe inference library that improves the maximum achievable serving throughput of Llama-3-8B by 1.2× on A100, 1.4× on L40S; and Qwen1.5-72B by 2.4× on A100, 3.5× on L40S, surpassing the leading industry solution TensorRT-LLM.

VILA: On Pre-training for Visual Language Models

CVPR 2024

(

)

VILA is a visual language model (VLM) pre-trained with interleaved image-text data at scale, enabling multi-image VLM. VILA is deployable on the edge.

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

ICML 2024

(

)

Quest is an effient long-context LLM inference framework that leverages query-aware sparsity in KV cache to reduce memory movement during attention and thus boost throughput.

Blog Posts

Block Sparse Attention

October 10, 2024

We introduce Block Sparse Attention, a library of sparse attention kernels that supports various sparse patterns, including streaming attention with token granularity, streaming attention with block granularity, and block-sparse attention. By incorporating these patterns, Block Sparse Attention can significantly reduce the computational costs of LLMs, thereby enhancing their efficiency and scalability. We release the implementation of Block Sparse Attention, which is modified base on FlashAttention 2.4.2.

TinyChat: Visual Language Models & Edge AI 2.0

March 3, 2024

Explore the latest advancement in TinyChat and AWQ – the integration of Visual Language Models (VLM) on the edge! The exciting advancements in VLM allows LLMs to comprehend visual inputs, enabling seamless image understanding tasks like caption generation, question answering, and more. With the latest release, TinyChat now supports leading VLMs such as VILA, which can be easily quantized with AWQ, empowering users with seamless experience for image understanding tasks.

TinyChat: Large Language Model on the Edge

September 6, 2023

Running large language models (LLMs) on the edge is of great importance. In this blog, we introduce TinyChat, an efficient and lightweight system for LLM deployment on the edge. It runs Meta's latest LLaMA-2 model at 30 tokens / second on NVIDIA Jetson Orin and can easily support different models and hardware.