SGLang

The AI revolution demands a new kind of infrastructure — and the AI Lab video series is your technical deep dive, discussing key engineering challenges and providing the blueprint you need to build the future of AI. For those who have struggled with crippling inference latency when serving a large language model, this video provides the technical guide for understanding how to begin optimizing. Crusoe Solutions Engineer Martin Cala dives deep into the architecture of high-throughput open-source LLM inference, explaining the core problem and how key innovations deliver massive performance gains. The challenge is clear: traditional methods waste huge amounts of GPU vRAM through memory fragmentation. The solution lies in a clever trick borrowed from operating systems: Paged Attention. In this episode, we cover: Paged attention: The core optimization that breaks the KV cache into uniform, fixed-size memory blocks to eliminate fragmentation and dramatically improve GPU utilization. vLLM + SGLang: A technical breakdown of the two leading open-source inference engines built on this principle, detailing when to use vLLM (high-throughput for dense models) and when SGLang excels (complex MoE architectures, function calling, and JSON parsing). The operational burden: Why managing these advanced, complex optimizations on purpose-built infrastructure is a major barrier for most companies. Chapters: 0:18 – The core problem: KV cache memory fragmentation 1:00 – Paged Attention: A game-changing OS trick 2:05 – vLLM: The high-throughput default for dense models 2:25 – SGLang: When to choose MoE and complex application runtimes 2:59 – Eliminating operational complexity with Crusoe Managed Inference For most companies, managing the complexity of paged attention, a cluster-wide KV cache, and resilient scaling is a huge operational burden. That's why we built Crusoe Managed Inference. We handle the infrastructure, the proprietary memory fabric, and the advanced cluster optimization, freeing your engineers to focus solely on building your product. Ready to stop optimizing infrastructure and start innovating? Crusoe Managed Inference: Learn how to access a cluster-wide KV cache and SLA-backed production deployments: https://www.crusoe.ai/cloud/managed-inference For AI engineers: Read the technical details of our proprietary platform: https://www.crusoe.ai/resources/blog/crusoe-memoryalloy-reinventing-kv-caching-for-cluster-scale-inference Launch blog: Get the full story on why we built Managed Inference: https://www.crusoe.ai/resources/blog/crusoe-managed-inference-optimize-performance-for-demanding-ai-workloads Start building today: Sign up for Crusoe Cloud and deploy your first workload: https://console.crusoecloud.com/request-foundry Don't forget to like this video and subscribe to the Crusoe channel for more episodes of AI Lab!

Do you want to learn how to serve models like DeepSeek and Qwen with SOTA speeds on launch day? SGLang is an open-source fast serving framework for LLMs and VLMs that generates trillions of tokens per day at companies like xAI, AMD, and Meituan. This workshop guides AI engineers who are familiar with serving models using frameworks like vLLM, Ollama, and TensorRT-LLM through deploying and optimizing their first model with SGLang, as well as providing guidance on when SGLang is the appropriate tool for LLM workloads. About Philip Kiely Philip Kiely leads Developer Relations at Baseten. Prior to joining Baseten in 2022, he worked across software engineering and technical writing for a variety of startups. Outside of work, you'll find Philip practicing martial arts, reading a new book, or cheering for his adopted bay area sports teams. About Yineng Zhang Yineng Zhang is a Software Engineer at Baseten Model Performance team. He is also a core developer of the SGLang project. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter 00:00 Introduction to LLM serving with SGLang 02:14 What is SGLang? 03:36 History of SGLang 06:49 Deploying Your First Model 13:01 Optimizing Performance with CUDA Graph Max Batch Size 24:19 Optimizing Performance with Eagle 3 Speculative Decoding 30:02 SGLang Community and Contributions 35:24 Invitations and Job Opportunities 36:52 Q&A