Promptfoo Integration with Litefuse Prompt Management

Promptfoo is an open-source LLM testing platform. It offers tools for red teaming, vulnerability scanning, and LLM evaluations, helping teams ensure the security and reliability of their LLM applications.

Integrate Promptfoo with Litefuse to take advantage of Litefuse’s prompt management features during your Promptfoo evaluations. By using this integration, you can easily reference and manage prompts directly within Litefuse, making your LLM testing process more efficient and organized.

With Litefuse Prompt Management, you can:

Update prompts without redeploying your application.
Track and revert to previous prompt versions.
Monitor and optimize prompt performance.
Integrate prompts seamlessly with your tools and applications.
Manage prompts via UI, SDKs, or API with minimal latency.

For more details, visit the Litefuse Prompt Management documentation.

Thanks to the team at Promptfoo for developing this integration (docs).

Quick Start Guide

Step 1: Set up Litefuse

Install the litefuse SDK: npm install langfuse
Visit Litefuse to create an account.
Create a new project and copy your Langfuse API keys.
Set the LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL environment variables as desired.

Step 2: Create a Litefuse Prompt

You can create, edit and manage your prompts via the Litefuse UI, SDKs, or API.

# Create a text prompt
langfuse.create_prompt(
    name="movie-critic",
    type="text",
    prompt="As a {{criticlevel}} movie critic, do you like {{movie}}?",
    labels=["production"],  # directly promote to production
    config={
        "model": "gpt-3.5-turbo",
        "temperature": 0.7,
        "supported_languages": ["en", "fr"],
    },  # optionally, add configs (e.g. model parameters or model tools) or tags
)
 
# Create a chat prompt
langfuse.create_prompt(
    name="movie-critic-chat",
    type="chat",
    prompt=[
      { role: "system", content: "You are an {{criticlevel}} movie critic" },
      { role: "user", content: "Do you like {{movie}}?" },
    ],
    labels=["production"],  # directly promote to production
    config={
        "model": "gpt-3.5-turbo",
        "temperature": 0.7,
        "supported_languages": ["en", "fr"],
    },  # optionally, add configs (e.g. model parameters or model tools) or tags
)

If you already have a prompt with the same name, the prompt will be added as a new version.

// Create a text prompt
await langfuse.createPrompt({
  name: "movie-critic",
  type: "text",
  prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
  labels: ["production"], // directly promote to production
  config: {
    model: "gpt-3.5-turbo",
    temperature: 0.7,
    supported_languages: ["en", "fr"],
  }, // optionally, add configs (e.g. model parameters or model tools) or tags
});
 
// Create a chat prompt
await langfuse.createPrompt({
  name: "movie-critic-chat",
  type: "chat",
  prompt: [
    { role: "system", content: "You are an {{criticlevel}} movie critic" },
    { role: "user", content: "Do you like {{movie}}?" },
  ],
  labels: ["production"], // directly promote to production
  config: {
    model: "gpt-3.5-turbo",
    temperature: 0.7,
    supported_languages: ["en", "fr"],
  }, // optionally, add configs (e.g. model parameters or model tools) or tags
});

If you already have a prompt with the same name, the prompt will be added as a new version.

Step 3: Reference Prompts in Promptfoo

Now you can use the prompts you created in Litefuse with Promptfoo. Use the langfuse:// prefix for your prompts in your Promptfoo configuration file, followed by the Litefuse prompt ID and version. For example:

prompts:
  - "langfuse://foo-bar-prompt:3"
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      # ...

Variables from your Promptfoo test cases will be automatically plugged into the Litefuse prompt as variables.

Parallel Testable Minds

Was this page helpful?

Support