Snorkel AI

Snorkel AI

Snorkel AI provides a platform for developing expert-level training data, benchmarks, and evaluation environments tailored for high-stakes, frontier AI applications. It enables teams to create and manage data programs that improve model and agent performance in specialized domains. The platform includes tools for data labeling, model fine-tuning, and creating synthetic data environments. Target users are AI researchers, data scientists, and ML engineers building advanced models.

Search
Visit website
Added on
Sep 22, 2026
Data-centric AITraining dataModel evaluationExpert labelingBenchmarks
Product information

Everything worth knowing about Snorkel AI

The complete picture — from everyday features to the technical detail builders and IT folks dig for.

Core features

What it actually does

Specialized Training Data Creation
Develop high-quality, domain-specific training data to fine-tune models for high-stakes, specialized tasks.
Benchmark & Evaluation Environments
Build custom benchmarks and evaluation suites to measure and validate model performance in real-world scenarios.
Data-Centric AI Platform
A platform for iterating on data, labels, and model evaluation rather than just model architectures.
Expert Data Labeling & Curation
Leverage expert human-in-the-loop processes to create gold-standard labels and datasets.
Technical capabilities
Programmatic Data Labeling
Use heuristic functions and labeling models to scale data annotation beyond manual effort.
Model Quality & Performance Analysis
Tools for error analysis, data slicing, and performance diagnostics to understand model weaknesses.
Must watch videos
OSWorld 2.0: Long-Horizon Computer-Use Benchmark | Snorkel AI Reading Group
50:51
Tutorial50:51
OSWorld 2.0: Long-Horizon Computer-Use Benchmark | Snorkel AI Reading Group
At our latest Snorkel AI Reading Group, Mengqi Yuan presents OSWorld 2.0, a benchmark of 108 long-horizon, real-world computer-use workflows spanning 31 self-hosted websites and professional desktop apps. Tasks average more than 300 agent steps and 27 scoring checkpoints, and 69.6% take a skilled human over an hour to complete, a sharp jump from OSWorld 1.0's roughly 30-step tasks. Can today's frontier agents handle the cross-source reasoning, dynamic pop-ups, streaming interaction, and implicit-state inference that real, messy computer-use work demands? The best system in the paper completes only 20.6% of tasks outright, though a Claude model released the day before this talk pushed partial credit above 60%. The talk covers what changed since OSWorld 1.0, the ten challenge phenomena built into the benchmark, four recurring agent failure modes, and the safety checkpoints woven into the task suite, from leaked API keys to destructive file deletions. Chapters 00:00 Introduction 02:59 What is OSWorld, and why OSWorld 1.0 saturated 04:17 Benchmark scale: 300–500 step tasks across multiple apps 05:01 Example task: the multi-app reimbursement workflow 07:50 Letting agents ask a simulated user for clarification 08:39 By the numbers: 31 self-hosted sites, 27 checkpoints per task 10:35 Ten challenge phenomena: cross-source reasoning to implicit state 14:00 Conflicting information: the concert-seat example 15:15 Tutorials, dynamic environments, and streaming tasks 17:40 Live showcase: dynamic pop-ups and streaming interaction 18:22 Self-hosting 31 websites for a stable environment 19:06 How tasks are sourced, designed, and verified 20:43 Frontier agent results: Claude vs. GPT 22:09 Domain-level differences between Claude and GPT 22:54 Four failure modes: information, perception, verification, memory 25:55 Safety checkpoints: leaked API keys and destructive actions 26:53 Q&A 49:52 Wrap-up and the Frontier Data Summit 🔗 Links 📄 Paper: https://arxiv.org/abs/2606.29537 🌐 Benchmark & Leaderboard: https://osworld-v2.xlang.ai/ | https://snorkel.ai/leaderboard/os-world-2-0/ 💻 GitHub: https://github.com/xlang-ai/OSWorld-V2 🐦 Mengqi Yuan on X: https://x.com/yuan_mengq43669 🔗 Mengqi Yuan on LinkedIn: https://www.linkedin.com/in/mengqi-yuan-ab850a323/ 📖 Blog post with full transcript: https://snorkel.ai/blog/osworld-2-0-why-computer-use-agents-fail-most-tasks 🎙️ Reading Group info: https://snorkel.ai/reading-group/ Previous Reading Group sessions: ▶️ Train-to-Test (T²) Scaling Laws: https://snorkel.ai/blog/train-to-test-scaling-laws-reading-group/ ▶️ Senior SWE-Bench: Evaluating Coding Agents: https://snorkel.ai/blog/senior-swe-bench-evaluating-coding-agents-like-senior-engineers/ ▶️ Agents' Last Exam: https://snorkel.ai/blog/agents-last-exam-reading-group/ ▶️ JudgmentBench: Rubric vs. Preference Evaluation: https://snorkel.ai/blog/judgment-bench-comparing-rubric-and-preference-evaluation-for-quality-assessment-legal-ai/ ▶️ Collaborative Gym: Human-Agent Collaboration: https://snorkel.ai/blog/collaborative-gym-a-framework-for-enabling-and-evaluating-human-agent-collaboration/ ▶️ Code World Models & AutoHarness: https://snorkel.ai/blog/code-world-models-and-autoharness-for-llm-agents/ ▶️ Olmix: Data Mixing for LM Development: https://snorkel.ai/blog/olmix-data-mixing-reading-group/
Continual Learning Bench: Do AI Systems Learn? | Snorkel AI Reading Group
36:07
Tutorial36:07
Continual Learning Bench: Do AI Systems Learn? | Snorkel AI Reading Group
At our latest Snorkel AI Reading Group, Parth Asawa presents Continual Learning Bench, the first expert-validated benchmark built to test whether LLM-based systems genuinely improve with experience. CL-Bench spans six domains, from software engineering and database querying to outbreak forecasting and strategic game-playing, and each task sequence hides a latent structure a stateful system can discover online and a stateless one cannot. Standard benchmarks treat every instance as independent, so they measure capability rather than learning. To separate the two, CL-Bench introduces a gain metric: stateful reward minus the same system's own stateless reward, controlling for instance difficulty and base model strength. The headline result was counterintuitive: naive in-context learning beat dedicated memory systems on the Pareto frontiers, and agents frequently overfit to the most recent observation instead of reusing what they knew. The talk covers the benchmark's three design criteria, how the gain metric isolates learning, and why native continual learning may need a different training stack. --- Chapters 00:00 Introduction 01:18 We talk about what models do, not how much they learn 01:53 How benchmarks are scored today 03:15 What learning should actually look like 03:36 Defining continual learning 04:34 Context, externalized memory, or parametric updates 05:21 How continual learning is evaluated today 07:55 Why you can't just chain existing benchmarks 09:24 Design criterion: headroom 09:48 Design criterion: shared latent structure 10:26 Design criterion: a learning mechanism 10:50 Why cumulative reward isn't enough 13:20 The gain metric: stateful minus stateless 15:05 Gain, reward, and cost on Pareto frontiers 15:27 Task walkthrough: database exploration 17:44 Concept drift and database migrations 19:18 The other five tasks in CL-Bench 1.0 20:24 Results: the leaderboard 21:03 Naive in-context learning wins the Pareto frontier 22:14 Stability and plasticity failure modes 25:03 The case for parametric methods 25:37 Rethinking the training stack from first principles 29:32 What's next for Continual Learning Bench 31:11 Q&A --- 🔗 Links 📄 Paper: https://arxiv.org/abs/2606.05661 🌐 Benchmark & Leaderboard: https://continual-learning-bench.com/ 💻 GitHub: https://github.com/pgasawa/continual-learning-bench 🐦 Parth Asawa on X: https://x.com/pgasawa 🔗 Parth Asawa on LinkedIn: https://www.linkedin.com/in/pgasawa/ 📖 Blog post with full transcript: https://snorkel.ai/blog/continual-learning-bench-reading-group/ 🎙️ Reading Group info: https://snorkel.ai/reading-group/ --- Previous Reading Group sessions: ▶️ Train-to-Test (T²) Scaling Laws — https://snorkel.ai/blog/train-to-test-scaling-laws-reading-group/ ▶️ Senior SWE-Bench: Evaluating Coding Agents — https://snorkel.ai/blog/senior-swe-bench-evaluating-coding-agents-like-senior-engineers/ ▶️ Agents' Last Exam — https://snorkel.ai/blog/agents-last-exam-reading-group/ ▶️ JudgmentBench: Rubric vs. Preference Evaluation — https://snorkel.ai/blog/judgment-bench-comparing-rubric-and-preference-evaluation-for-quality-assessment-legal-ai/ ▶️ Collaborative Gym: Human-Agent Collaboration — https://snorkel.ai/blog/collaborative-gym-a-framework-for-enabling-and-evaluating-human-agent-collaboration/ ▶️ Olmix: Data Mixing for LM Development — https://snorkel.ai/blog/olmix-data-mixing-reading-group/
Train-to-Test (T²) Scaling Laws | Snorkel AI Reading Group
36:38
Tutorial36:38
Train-to-Test (T²) Scaling Laws | Snorkel AI Reading Group
At our latest Snorkel AI Reading Group, Nicholas Roberts presents Train-to-Test (T²) Scaling Laws — work heading to COLM 2026 that asks what happens to compute-optimal pre-training once you account for the cost of reasoning at inference time. Chinchilla trades model size against training tokens and lands everyone on roughly 20 tokens per parameter, but it says nothing about inference. T² adds a power law in k, holds inference cost constant across model sizes, and concludes you should overtrain dramatically. Mid-paper, Liquid AI released LFM 2.5 — 350M parameters trained on 28 trillion tokens, about 80,000 tokens per parameter — landing almost exactly where T² predicted. The T²-selected checkpoint almost always significantly outperforms a Chinchilla-optimal one at matched inference budget, and the result survives fine-tuning and SFT. The talk covers pre-training scaling laws from first principles, why parallel sampling is the tractable test-time method to analyze, and a Q&A that pushes hard on the modeling assumptions. Chapters 00:00 Introduction 00:28 Roadmap 01:27 What are scaling laws? 03:25 Chinchilla and the 20-tokens-per-parameter rule 05:35 But everyone is doing reasoning now 06:14 Test-time scaling techniques 07:06 Parallel sampling and pass@k 08:15 Train-to-Test (T²) scaling 09:43 Adding a third term: a power law in k 10:24 Inference-cost matching across model sizes 12:03 Why k can go inside a model of the loss 12:45 A second model: predicting accuracy directly 13:10 Two models — loss and pass@k 14:43 The forecast looks radically different 15:00 The catch: we might run out of data 15:34 Liquid AI's LFM 2.5 lands mid-paper 16:06 80,000 tokens per parameter 17:45 How good is the scaling model? 18:38 T² vs. Chinchilla-optimal, head to head 19:13 What happens to the IsoFLOP curves 20:14 The post-training objection 21:34 Results hold after fine-tuning and SFT 22:21 The first end-to-end scaling law 23:01 Q&A 🔗 Links 📄 Paper: https://arxiv.org/abs/2604.01411 📰 VentureBeat coverage: https://venturebeat.com/orchestration/train-to-test-scaling-explained-how-to-optimize-your-end-to-end-ai-compute-budget-for-inference 👤 Nicholas Roberts: https://nick11roberts.science/ 🐦 Nicholas Roberts on X: @nick11roberts 📖 Blog post with full transcript: https://snorkel.ai/blog/train-to-test-scaling-laws-reading-group/ 🎙️ Reading Group info: https://snorkel.ai/reading-group/ Previous Reading Group sessions: ▶️ Senior SWE-Bench: Evaluating Coding Agents — https://snorkel.ai/blog/senior-swe-bench-evaluating-coding-agents-like-senior-engineers/ ▶️ Agents' Last Exam — https://snorkel.ai/blog/agents-last-exam-reading-group/ ▶️ JudgmentBench: Rubric vs. Preference Evaluation — https://snorkel.ai/blog/judgment-bench-comparing-rubric-and-preference-evaluation-for-quality-assessment-legal-ai/ ▶️ Collaborative Gym: Human-Agent Collaboration — https://snorkel.ai/blog/collaborative-gym-a-framework-for-enabling-and-evaluating-human-agent-collaboration/ ▶️ Code World Models & AutoHarness — https://snorkel.ai/blog/code-world-models-and-autoharness-for-llm-agents/ ▶️ Olmix: Data Mixing for LM Development — https://snorkel.ai/blog/olmix-data-mixing-reading-group/

Who uses it

Real use cases, no hype
Training Data Curation
Build high-quality, labeled datasets for supervised ML without massive manual annotation.
LLM Fine-Tuning Data
Generate specialized instruction-following or domain-specific examples for fine-tuning large language models.
Domain-Specific AI Models
Rapidly develop data for niche fields like finance, legal, or healthcare where experts are scarce.
Data-Centric AI Iteration
Systematically improve model performance by focusing on data quality and coverage over model architecture.
High-Stakes Prediction
Create reliable datasets for critical applications where labeling errors have significant consequences.
What's great
  • Dramatically reduces manual labeling time and cost
  • Enables rapid iteration on labeling functions for complex domains
  • Clear data versioning and lineage tracking
  • Specializes in programmatic data development for high-stakes AI
Technical strengths
  • Core framework uses weak supervision with labeling functions (LFs)
  • Generative model aggregates noisy LF outputs into probabilistic labels
  • Supports fine-tuning data generation via LLM-based LFs
Where it falls short
  • Steep learning curve for non-technical domain experts
  • Pricing is opaque and enterprise-focused
  • Overkill for simple, straightforward labeling tasks
Technical limitations
  • Requires engineering effort to design effective labeling functions
  • Dependency on the underlying generative model's assumptions
For technical folks

The deep-dive specs

FAQ

Includes technical Q&A
Snorkel is based on 'data-centric AI.' Instead of hand-labeling massive datasets, you programmatically label data using 'labeling functions'—simple rules or heuristics written by domain experts. A generative model then combines these noisy signals to produce high-quality probabilistic labels for your training data.

Traffic Insights

Monthly Visits

—

November 2026

Growth Rate

0%

vs last month

Dominance

0%

in Search

Top Country

—

Top Countries Breakdown

—

Growth Trend

Stable

Traffic has remained stable.

Pricing overview
Paid Pricing

Snorkel AI Pricing

Snorkel AI uses a paid pricing structure. Real-time rates, custom enterprise quotas, and plan details are managed directly on their official platform.

Check Pricing on Snorkel AI
Reviews

What the internet actually thinks

Data refreshed weekly. No paid placements.

Trustpilot
Not reviewed yet
G2
G2
Not reviewed yet
C
Capterra
Not reviewed yet
Product Hunt
Not reviewed yet
COMMUNITY COMMENTS

Leave a Comment

Help others make informed decisions. Your honest feedback shapes the community's understanding of this tool.

Tips for commenting

  • •Be specific about features you used
  • •Share real use cases and results
  • •Mention both pros and cons
  • •Keep it honest and constructive
to leave a comment

Your comment is published immediately.

Filter:
Sort:

No comments yet

Be the first to share your experience with this tool.

Alternatives

Not quite the right fit?

Here's what else is out there.