How does the 10k probe limit on red-teaming work?

A probe is a single attack request. The free tier allows 10k per month. Enterprise customers can purchase higher limits. This is because some dynamic attack plugins require LLM inference, which has a cost.

Can I run promptfoo without sending data to your servers?

Yes. The core software runs entirely on your machine. It only communicates directly with the LLM API providers you configure (e.g., OpenAI). No data is sent to Promptfoo unless you use the managed Enterprise cloud service.

What LLM providers and models are supported?

Promptfoo is provider-agnostic. It supports OpenAI, Anthropic, Azure, Google, HuggingFace, and any OpenAI-compatible API. You can test proprietary models, open-source models via Ollama or HuggingFace, or custom endpoints.

How does red-teaming integrate into a CI/CD pipeline?

You can run `npx promptfoo@latest redteam scan` as a step in your GitHub Action or Jenkinsfile. It will output a vulnerability report and can be configured to fail the build based on severity thresholds.

Can I write custom red-team plugins or attack strategies?

Yes. The plugin system allows you to define custom attack flows and generators using JavaScript/TypeScript. This lets you tailor tests to your specific business logic and threats.

Directory/Search/AI Security/Promptfoo

Promptfoo

Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. It enables developers to build reliable prompts and models with benchmarks, secure apps via automated red teaming and pentesting, and speed up evaluations with caching and concurrency. Targeted at AI developers and security teams, it integrates with CI/CD pipelines and supports all major LLM providers. Key capabilities include custom attack generation for vulnerabilities like prompt injections and jailbreaks, real-time threat intelligence from a large user community, and comprehensive evaluation reports. It's designed for test-driven LLM development, replacing trial-and-error with systematic testing.

Visit website

Added on

Sep 22, 2026

Open-source CLILLM Red TeamingPrompt EvaluationCI/CD IntegrationVulnerability ScanningYAML ConfigurationSOC2 Certified

Product information

Everything worth knowing about Promptfoo

The complete picture — from everyday features to the technical detail builders and IT folks dig for.

Core features

What it actually does

Automated Red Teaming

Simulates real users to uncover application-specific vulnerabilities like prompt injections, jailbreaks, data leaks, and business rule violations.

LLM Evaluation & Benchmarking

Build and test reliable prompts, models, and RAG pipelines with custom benchmarks and matrix views for side-by-side comparison.

CI/CD Integration & Dev Workflow

Connects to GitHub, GitLab, Jenkins, and other CI/CD pipelines to automate security testing and remediation in pull requests.

Continuous Monitoring & Dashboards

Provides a centralized dashboard for tracking vulnerabilities, remediation progress, and compliance status across projects.

Technical capabilities

Open-Source CLI & Library

Available as a command-line tool and Node.js library for local, private execution with live reloading and caching.

Declarative Test Configuration

Defines evaluations and red team tests using simple YAML configurations without complex notebooks.

Must watch videos

Who uses it

Real use cases, no hype

LLM Security Red Teaming

Automate the discovery of vulnerabilities like prompt injection, jailbreaks, and data leaks in your AI agents and RAG systems.

Prompt & Model Evaluation

Systematically test and compare prompt variations, model outputs, and RAG pipelines with custom assertions and metrics.

CI/CD Security Integration

Integrate LLM vulnerability scanning directly into your GitHub Actions or Jenkins pipelines to catch regressions early.

Compliance & Risk Auditing

Generate detailed reports for frameworks like FINRA, SOC2, or ISO 27001 to demonstrate AI system security and fairness.

Guardrail Validation

Test the effectiveness of your real-time jailbreak prevention and content filtering systems against adversarial attacks.

MCP & Agent Security

Securely proxy and test Model Context Protocol communications and tool-using agents for insecure usage patterns.

What's great

Strong open-source core with a large, active community.
Excellent developer experience: fast, CLI-first, with live reloads and caching.
Comprehensive red-teaming suite that simulates real-world attacks.
Simple, declarative YAML-based test configuration.

Technical strengths

Language-agnostic: works with Python, JS, or any API-based language.
Extensible plugin system for custom red-team attack strategies.
Runs locally for complete data privacy and no vendor lock-in.

Where it falls short

Advanced features like continuous monitoring require a paid Enterprise plan.
Usage limits on the free tier for advanced red-teaming probes.
Learning curve for advanced configuration and custom plugin development.

Technical limitations

Enterprise SSO and granular permissions are not available in the community edition.
Reporting and dashboard features are limited without the centralized Enterprise UI.

For technical folks

The deep-dive specs

FAQ

Includes technical Q&A

The Community version is a powerful, open-source CLI for local testing and red-teaming. Enterprise adds a centralized cloud dashboard, team collaboration, SSO, higher usage limits, and continuous monitoring.