Rules Promptfoo

Mon, 01 Jan 0001 00:00:00 +0000

rules_promptfoo runs repository skills through a pinned Promptfoo CLI. It keeps dependency resolution reproducible in Bazel while deliberately running model calls only at test runtime.

Setup

Add the module and its skill-provider dependency:

bazel_dep(name = "rules_promptfoo", version = "<VERSION>")
bazel_dep(name = "rules_skill", version = "<VERSION>")

The development checkout uses sibling local_path_override declarations. Published releases must replace those local overrides with registry versions.

Evaluate a skill

load("@rules_promptfoo//promptfoo:defs.bzl", "promptfoo_test")

promptfoo_test(
    name = "answer_question_eval",
    config = "promptfooconfig.yaml",
    skills = ["//projects/agents/skills/answer-question:skill"],
    env_inherit = [
        "CODEX_HOME",
        "CODEX_PATH_OVERRIDE",
    ],
    reuse_codex_login = True,
)

The runner physically copies each selected SkillInfo.files_by_path bundle to an isolated Bazel test-temporary workspace/.agents/skills/<name>/ tree. Point the Codex SDK provider at the exported workspace and, when using a compatible host Codex executable, at the exported executable path:

Rules Promptfoo Gazelle

Mon, 01 Jan 0001 00:00:00 +0000

rules_promptfoo_gazelle generates required, offline promptfoo_validate_test targets for Promptfoo configurations below an evals/ directory. It never creates an evals subpackage and never generates live promptfoo_test targets.

Add rules_promptfoo as a normal dependency and this generator as a development dependency:

bazel_dep(name = "rules_promptfoo", version = "<VERSION>")
bazel_dep(
    name = "rules_promptfoo_gazelle",
    version = "<VERSION>",
    dev_dependency = True,
)

For one-pass skill package creation, also add rules_skill as a normal dependency and rules_skill_gazelle as a development dependency:

bazel_dep(name = "rules_skill", version = "<VERSION>")
bazel_dep(
    name = "rules_skill_gazelle",
    version = "<VERSION>",
    dev_dependency = True,
)

Add both public language targets to the repository’s custom Gazelle binary:

Llm_evals on Alwaldend

Rules Promptfoo

Setup

Evaluate a skill

Rules Promptfoo Gazelle