Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for a more robust multi-prompt evaluation is challenging, limiting its adoption in practice.

To address this, we introduce PromptSuite, a framework that enables the automatic generation of various prompts. PromptSuite is flexible – working out of the box on a wide range of tasks and benchmarks. It follows a modular prompt design, allowing controlled perturbations to each component, and is extensible, supporting the addition of new components and perturbation types.

Through a series of case studies, we show that PromptSuite provides meaningful variations to support strong evaluation practices. It is available through both a Python API and a user-friendly web interface.

Get started: https://github.com/eliyahabba/PromptSuite

We've made our benchmark results publicly available on Hugging Face! The dataset contains prompt variations and model outputs in DOVE format across multiple tasks and models.

from datasets import load_dataset

# Load MMLU anatomy with GPT-4o-mini
dataset = load_dataset(
    "nlphuji/PromptSuite",
    data_files="GPT-4o-mini/en/2_shots/mmlu:dataset=mmlu.anatomy*.parquet"
)

# Load all tasks for a model
all_tasks = load_dataset(
    "nlphuji/PromptSuite",
    data_files="GPT-4o-mini/en/2_shots/*.parquet"
)

The dataset includes metadata for prompt variations, model responses, and evaluation metrics. Perfect for studying prompt sensitivity and robustness across different tasks and models.

Table 1: Overview of the perturbation types supported by PromptSuite. The "Applicable Components" column specifies which prompt components each perturbation can be applied to. For example, paraphrasing is applicable to the instruction component.

📦 Installation

Install PromptSuite from PyPI:

pip install promptsuite

Or install from source:

git clone https://github.com/eliyahabba/PromptSuite.git
cd PromptSuite
pip install -e .

🚀 Basic Usage

Generate prompt variations with just a few lines of code:

from promptsuite import PromptSuite
import pandas as pd

# Initialize
ps = PromptSuite()

# Load data
data = [{"question": "What is 2+2?", "answer": "4"}]
ps.load_dataframe(pd.DataFrame(data))

# Configure template
template = {
    'instruction': 'Please answer the following questions.',
    'prompt format': 'Q: {question}\nA: {answer}',
    'instruction variations': ['paraphrase_with_llm'],
    'prompt format variations': ['format structure'],
}
ps.set_template(template)

# Generate variations
ps.configure(variations_per_field=3, api_platform="OpenAI", model_name="gpt-4o-mini")
variations = ps.generate(verbose=True)

# Export results
ps.export("output.json", format="json")

🎯 Advanced Examples

Sentiment Analysis

import pandas as pd
from promptsuite import PromptSuite

data = pd.DataFrame({
    'text': ['I love this movie!', 'This book is terrible.'],
    'label': ['positive', 'negative']
})

template = {
    'instruction': 'Classify the sentiment',
    'instruction variations': ['paraphrase_with_llm'],
    'prompt format': 'Text: {text}\nSentiment: {label}',
    'text': ['typos and noise'],
}

ps = PromptSuite()
ps.load_dataframe(data)
ps.set_template(template)
ps.configure(variations_per_field=3, max_variations_per_row=2)
variations = ps.generate(verbose=True)

Multiple Choice with Few-shot

template = {
    'instruction': 'Answer the question:\nQuestion: {question}\nAnswer: {answer}',
    'instruction variations': ['paraphrase_with_llm'],
    'question': ['semantic'],
    'gold': 'answer',
    'few_shot': {
        'count': 2,
        'format': 'same_examples__synchronized_order_variations',
        'split': 'train'
    }
}

ps = PromptSuite()
ps.load_dataframe(qa_data)
ps.set_template(template)
ps.configure(variations_per_field=2)
variations = ps.generate(verbose=True)

💾 Export Formats

# JSON - Full data with metadata
ps.export("output.json", format="json")

# CSV - Flattened for spreadsheets
ps.export("output.csv", format="csv")

# TXT - Plain prompts only
ps.export("output.txt", format="txt")

This project is licensed under the MIT License. See the LICENSE file for details.