01
Baseline behavior
Capture representative scenarios and define expected behavior before making a change.
The problem
AI behavior can regress without a traditional code failure.
A prompt change, model update, retrieval change, or application logic change can alter output quality. A repeatable baseline-and-compare workflow helps teams detect those changes before they become release problems.
02
Repeatable evaluation
Run the same regression suite after prompt, model, retrieval, or application changes.
03
Evidence-driven QA
Use datasets, automated results, reports, and CI evidence to investigate regressions.
Free MVP
Everything you need to start.
A practical collection of code, datasets, examples, templates, and documentation for building an AI regression workflow.
CSV
AI regression datasets
CSV-based evaluation data for representative AI testing scenarios.
PY
Python + pytest
Reusable regression testing examples built around Python and pytest.
PF
Promptfoo example
A starting point for structured LLM evaluation workflows using Promptfoo.
EV
Evaluation scorecard
Evaluation criteria and a scorecard approach for comparing AI response quality.
CI
CI workflow
GitHub Actions workflow support for repeatable automated regression execution.
DOC
Templates & docs
Reusable templates and documentation that can be adapted to project-specific testing.
Workflow
From baseline to release decision.
Use the same evaluation suite to compare AI behavior before and after a change.
01
Create a dataset
Represent important AI use cases.
02
Establish a baseline
Define expected behavior and quality thresholds.
03
Run the suite
Execute the same checks after changes.
04
Compare & investigate
Identify regressions before release.
Evidence
See the testing workflow in action.
The repository includes execution evidence and a self-contained HTML test report example.
Built for
QA teams working with AI.
Use the starter kit as a foundation for your own AI/LLM regression strategy.
✓ QA Engineers testing AI-powered applications
✓ SDETs building automated regression workflows
✓ Automation Engineers using Python & pytest
✓ Teams experimenting with LLM, RAG & prompts
Help improve the MVP
Tried the Free MVP?
Your feedback helps improve the starter kit, identify missing features, and shape future versions of the product.
Takes about 1–2 minutes · Feedback is optional
COMING SOON
From Free MVP to a deeper Pro edition.
The current MVP is the free starting point. Future Pro development may add deeper evaluation assets, advanced regression workflows, expanded examples, production-oriented templates, and additional documentation.
Start here
Explore the Free MVP.
Clone the repository, install the dependencies, run the examples, and adapt the workflow to your application.
FAQ
Common questions.
Is the MVP free?
Yes. The current starter kit is available as a free MVP through the public GitHub repository.
What technologies are included?
The project includes Python, pytest, Promptfoo examples, evaluation datasets, templates, documentation, and GitHub Actions.
Can I adapt it for my own AI application?
Yes. The kit is designed as a practical starting point that can be adapted to project-specific AI regression scenarios.
How can I provide feedback?
You can use the feedback form on this page. It takes approximately 1–2 minutes and helps improve future versions of the starter kit.
Is the Pro edition available now?
No. The Pro edition is planned for future development and is not being sold from this page.