Dagster

You've got four Databricks workspaces, Kafka streams, Fivetran syncs, legacy databases and zero unified visibility across any of them. Finance needs to know if upstream customer data is fresh. Marketing depends on sales data that lives in a completely different workspace. When Kafka goes down, nobody downstream knows their dashboards are broken. This is what most Databricks deployments look like at scale. Siloed workspaces, no cross-domain dependency graph, no single pane of glass. In this video, I walk through a full working example that brings 21 assets across four Databricks workspaces, two Kafka topics, two Fivetran connectors, and two legacy databases into a single Dagster asset graph — with full cross-domain lineage from raw ingestion through analytics output. You'll see: → How to use the DatabricksJobOrchestrator component to configure each workspace with YAML (no Python boilerplate) → Sensor-driven Kafka ingestion with offset-based deduplication → Cross-workspace dependency tracking so downstream teams know when upstream data is stale → How to layer on freshness policies, asset checks, and alerting across the entire graph Clone the project and follow along: https://docs.dagster.io/examples/full-pipelines/multi-workspace-databricks Dagster Slack community: https://dagster.io/slack Try Dagster+ free: https://dagster.plus/signup Chapters: 0:00 — The problem: siloed Databricks workspaces 0:25 — Data sources: Kafka, Fivetran, legacy databases 1:00 — Project structure and the DatabricksJobOrchestrator component 2:00 — Workspace configuration with YAML 3:00 — Core assets: Kafka, Fivetran, legacy federation 3:30 — Launching Dagster and viewing the full asset graph 4:00 — Sensor-driven Kafka ingestion 5:00 — Layering on freshness policies, asset checks, and alerting 6:00 — Role-based access control with Dagster+ 6:30 — Recap: what a data control plane looks like

0:00 – Intro 2:30 – What is multi-tenancy and why it matters 4:00 – The Titanic analogy: what happens when boundaries fail 5:00 – Centralized vs. embedded vs. hub-and-spoke data teams 7:15 – AI/LLM workloads and the multi-tenancy problem 9:00 – When multi-tenancy is (and isn't) the right call 10:15 – Dagster's "meatball sub" architecture: code locations, definitions, and resource scoping 12:30 – Common isolation patterns: team-based, client-based, domain-based, environment-based 14:15 – Case studies: StashAway, Magenta, and Ida 17:15 – Brooklyn Data client stories: financial services migrations, conglomerate reporting, LLM-powered document extraction 27:00 – Live demo: multi-tenant Dagster project with cross-code-location sensors and local LLM resources 32:00 – Key takeaways for platform teams and practitioners 36:30 – Q&A: partitions & multi-tenancy, external client permissions, experiment vs. production isolation, PII boundaries, RBAC in Dagster Plus, and more Resources mentioned: Dagster Case Studies: https://dagster.io/customers Dagster Docs – Code Locations: https://docs.dagster.io/guides/build/projects/workspaces/creating-workspaces Dagste+ RBAC: https://docs.dagster.io/deployment/dagster-plus/authentication-and-access-control Dagster Skills for AI-assisted development: https://github.com/dagster-io/skills Dagster Compass:https://compass.dagster.io/ Brooklyn Data Co.: https://www.brooklyndata.co/