Modal

Agents are only as useful as the tools they can call. But the way most MCP servers work today, loading every tool into the context window on every request, makes models slower and harder to build on. In this webinar, Adam Azzam, member of product staff at Modal and maintainer of FastMCP, walks through what's actually broken with MCP tool calling and introduces Code Mode: a server-side approach where agents write programs and execute them in isolated Modal Sandboxes, instead of chaining serial tool calls one at a time. *The problem with how MCP works today* When a client connects to an MCP server, it typically dumps the entire tool manifest into the context window. Thirty-five tools from a single server means roughly a 20,000-token penalty before a user types a word. Add Gmail, Slack, and Linear together and you're looking at a fixed 100,000-token tax just to start a conversation. Models don't perform well under those conditions. Dexter Horthy calls it the "dumb zone." Serial execution makes it worse. Even with parallel tool calling, models tend to lazily parallelize a handful of calls and then ask if they should continue. Asking an agent to create issues for 40 open tickets means sitting through 40 sequential round trips. It's a bad experience and a non-starter for production agents. *What Code Mode does* Instead of injecting 35 tools upfront and calling them one at a time, Code Mode reduces any tool catalog to three things the client ever sees: a search tool, a schema fetch, and a single execution tool. The agent writes a program, sends it to the server, and the server runs it in a Modal Sandbox. One round trip instead of forty. The backend can have tens of thousands of tools; clients never pay for it. The sandbox is isolated, has no network egress, enforces timeouts, and boots in about 500ms. Fast enough that it doesn't register in a normal request-response cycle. *FastMCP and how it fits together* FastMCP is a Python framework for building MCP servers. It ships with two main concepts: providers (where tools come from) and transforms (how you filter, rename, or reshape them before clients see them). Code Mode is implemented as a transform. You take your existing MCP server, add the Code Mode transform with a sandbox provider, and your clients go from seeing 35 tools to seeing 3. The rest happens server-side. Deploying it on Modal is one step. The integration handles the stdio bridge between the host server and the sandbox automatically. *Links* FastMCP: https://github.com/jlowin/fastmcp Modal Sandboxes: https://modal.com/docs/guide/sandbox Modal free tier: https://modal.com *Timestamps* 0:00 - Introductions 4:35 - FastMCP Project History & Community Contributions 9:10 - Agent Orchestration: Separating the Control Plane from the Data Plane 11:20 - Code Walkthrough: Ejecting FastMCP into an ASGI App 15:51 - Token Tax & Entering the LLM "Dumb Zone" 20:32 - The 3 Antagonists of Traditional MCP User Experience 25:21 - The History of Code Mode: Client-Side vs. Server-Side Execution 32:45 - How the Sandbox Works: Isolation, Timeouts, and Zero Network Egress 35:35 - Live Performance Comparison: Serial Tool Calls vs. Sandbox Program Execution 42:03 - Q&A: Best Practices for Minimizing Agent Cold Starts 43:53 - Q&A: Using Open-Source Local Models vs. Claude for Function Calling 45:17 - Q&A: How Modern Clients Strategically Load Tools 49:43 - Q&A: Does FastMCP Have Durable Execution Built-In? 54:46 - Q&A: Replacing Multi-Tool Latency with Native Python/Bash in a Sandbox

Building real-time, low-latency Voice AI applications requires dealing with infrastructure limits, strict latency budgets, and unpredictable user traffic. In this webinar, Modal forward deployed engineer Ben breaks down how to architect and deploy production-ready voice AI apps on Modal's AI infrastructure. From speech-to-text (STT) and text-to-speech (TTS) pipelines to handling voice agents, this breakdown covers the engineering patterns required to optimize for speed, quality, and compute costs. If you are building conversational voice bots, self-hosting open-source speech models, or scaling real-time GPU workloads, this talk breaks down the infrastructure challenges and the engineering solutions that work in production. *The Core Idea* Voice AI applications are sensitive to latency and experience fluctuating user demand. Relying solely on rigid, over-provisioned GPU hardware or multi-tenant proprietary APIs can lead to high costs, poor service quality, or severe P99 latency spikes. - Decouple the pipeline: Deploy your speech-to-text, LLM, and text-to-speech layers as independent serverless functions that auto-scale separately based on their specific concurrency limits. - Utilize GPU process snapshotting: Capture state checkpoints to speed up model startup times. - Interleave audio and compute workflows: Use streaming patterns, low-cost warm CPU nodes, and asynchronous audio execution to mask cold starts completely. The result: - *Massive Cost Reductions:* High-throughput batch transcription pipelines that run at a fraction of the cost of proprietary alternatives. - *Sub-Second Round Trips:* Responsive real-time voice agents optimized to minimize the time-to-first-speech. - *True Serverless Scaling:* Infrastructure that scales up instantly to handle peak usage bursts and tears down to zero when idle. *What You'll Learn* *The Core Components of Voice AI Pipelines* - How Speech to Text (STT) and Text to Speech (TTS) form the foundation of any voice system. - The role of Voice Activity Detection (VAD) and Diarization (speaker identification) in production audio processing. - Why conversational interfaces require distinct turn-taking models to differentiate between a brief user pause and a completed thought. *Optimizing Batch Audio Workloads* - How to build multi-level batching pipelines that scale horizontally across cloud GPUs simultaneously. - Why processing audio in a parallel forward pass out-competes traditional API structures. *Continuous Batching Limitations for Speech* - Why popular continuous batching architectures (like vLLM and SGLang) work natively for LLMs but do not yet cleanly exist as general-purpose engines for voice models. *How Modal Minimizes Latency & Cold Starts* - *GPU Process Snapshots:* Drops native model initialization and engine setup times (which normally range from 25 to 90 seconds) down to under 10 seconds. - *Warm CPU Buffering:* Keeping lightweight CPU nodes (like the frontend server and bot runner) continuously warm for pennies an hour ensures instant initial contact. - *Canned Intro Execution:* Instantly playing a localized, pre-recorded introduction audio file right upon connection to cleanly mask background model container startup times. - *First-Sentence Streaming:* Pulling the very first generated sentence from a streaming LLM output and sending it straight to the TTS engine immediately, allowing audio playback to start while subsequent sentences are still being processed down-line. *Key Performance Insights* - *Sort Batches by Audio Duration:* When batching audio on a GPU, the inference call will not complete until its longest element finishes. Sorting files by duration ensures uniform lanes, eliminating dead time and maximizing throughput. - *Isolated Scaling Parameters:* Because STT, LLMs, and TTS systems have vastly different concurrent throughput capacities, running them as individual microservices keeps bottlenecks isolated and prevents systemic failure. *Architecture Highlights* - *FastAPI Frontends:* Lightweight HTTP/WebRTC handshaking servers that dynamically spin up single-use, dedicated conversation containers on demand. - *Modular Agent Ecosystems:* Seamlessly working alongside leading real-time orchestration frameworks like PipeCat and LiveKit. - *Self-Hosted Model Tech Stack:* Strategies for utilizing engines like vLLM, SGLang, Triton, or Nemotron for self-hosting open-source variants. - *Self-Hosted Proprietary Deployments:* Step-by-step validation for hosting enterprise solutions (such as DeepGram) natively within your own dedicated serverless cluster.