title: AI Natural Language Tests emoji: 🧪 colorFrom: blue colorTo: green sdk: gradio sdk_version: "6.12.0" python_version: "3.10" app_file: app.py pinned: false

Try It Live

ai-natural-language-tests on Hugging Face Spaces — Try the platform directly in your browser without installation.

Product Preview

Generate tests from plain-English requirements, inspect workflow logs, review structured output, and download the generated spec from a single interface.

Table of Contents

Overview

The platform translates natural language requirements into executable E2E tests for:

flowchart LR
  A[AI-Powered E2E Test Generation]
  A --> C[Cypress<br/>.cy.js<br/>Traditional and prompt-powered]
  A --> P[Playwright<br/>.spec.ts<br/>TypeScript async/await]
  A --> W[WebdriverIO<br/>.spec.js<br/>Mocha with Jest-like expect]

  style A fill:#e3f2fd,color:#333333,stroke:#666666
  style C fill:#c8e6c9,color:#333333,stroke:#666666
  style P fill:#ffcdd2,color:#333333,stroke:#666666
  style W fill:#ffe0b2,color:#333333,stroke:#666666

It supports both local engineering workflows and automated pipeline execution. The generator uses contextual data from live HTML analysis and historical pattern matching to produce stable, maintainable test assets.

Quick Start (5 Minutes)

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

Create .env and set at least one provider key:

OPENAI_API_KEY=your_key

Generate and run one Playwright test:

python qa_automation.py "Test login with valid credentials" --url https://the-internet.herokuapp.com/login --framework playwright --run

Business Value

[!NOTE]

• Reduces manual test authoring effort and onboarding time.
• Standardizes generated test structure across teams.
• Improves reuse through vector-based pattern memory.
• Supports enterprise delivery with CI/CD and Docker workflows.
• Enables faster root-cause diagnosis using AI-assisted failure analysis.

Core Capabilities

Architecture

graph TB
    subgraph "User Input"
        A[Natural Language<br/>Requirements]
        B[URL/HTML Data<br/>--url flag]
    C[CLI Requirement Text<br/>one or more prompts]
    end

    subgraph "AI & Workflow Engine"
        D[LangGraph Workflow<br/>5-Step Process]
        E[Multi-Provider LLM<br/>OpenAI / Anthropic / Google]
        F[Vector Store<br/>Pattern Learning<br/>FAISS + SQLite]
    end

    subgraph "Framework Generation"
        G{Cypress Framework}
        H{Playwright Framework}
        W{WebdriverIO Framework}
        I[Cypress Tests<br/>.cy.js files<br/>Traditional & cy.prompt&#40;&#41;]
        J[Playwright Tests<br/>.spec.ts files<br/>TypeScript]
        X[WebdriverIO Tests<br/>.spec.js files<br/>Mocha + expect]
    end

    subgraph "Execution & Analysis"
        K[Cypress Runner<br/>npx cypress run]
        L[Playwright Runner<br/>npx playwright test]
        M[AI Failure Analyzer<br/>--analyze flag<br/>Multi-Provider LLM]
        P[WebdriverIO Runner<br/>npx wdio run]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> D
    D --> G
    D --> H
    D --> W
    G --> I
    H --> J
    W --> X
    I --> K
    J --> L
    X --> P
    K --> M
    L --> M
    P --> M

    style D fill:#e3f2fd,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#fff3e0,color:#333333,stroke:#666666
    style G fill:#c8e6c9,color:#333333,stroke:#666666
    style H fill:#ffcdd2,color:#333333,stroke:#666666
    style W fill:#ffe0b2,color:#333333,stroke:#666666

Workflow

flowchart TD
    A[Start: User Input<br/>Requirements + Framework] --> C[Step 2: Fetch Test Data<br/>Analyze URL/HTML<br/>Extract Selectors<br/>Generate Fixtures]
    C --> D[Step 3: Search Similar Patterns<br/>Query Vector Store<br/>Find Matching Test Patterns<br/>From Past Generations]
    D --> E[Step 4: Generate Tests<br/>Use AI + Patterns<br/>Create Framework-Specific Code<br/>Cypress, Playwright, or WebdriverIO]
    E --> H[HITL Approval<br/>Optional --approve before save]
    H --> F[Step 5: Run Tests<br/>Execute via Framework Runner<br/>Optional --run flag]
    F --> R[Replay Snapshot<br/>--list-html-replays / --replay-html-analysis]
    R --> G[End: Tests Executed<br/>Ready for CI/CD]

    style A fill:#e1f5fe,color:#333333,stroke:#666666
    style C fill:#c8e6c9,color:#333333,stroke:#666666
    style D fill:#ffcdd2,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#e8f5e8,color:#333333,stroke:#666666
    style G fill:#f3e5f5,color:#333333,stroke:#666666

Generation follows a deterministic five-step flow:

Technology Stack

Repository Structure

ai-natural-language-tests/
|-- cypress/
|   |-- e2e/
|   |   |-- generated/
|   |   `-- prompt-powered/
|   `-- fixtures/
|-- tests/
|   `-- generated/
|-- webdriverio/
|   `-- tests/
|       `-- generated/
|-- prompt_specs/
|-- vector_db/
|-- qa_automation.py
|-- qa_config.py
|-- qa_runtime.py
|-- qa_workflow.py
|-- cypress.config.js
|-- playwright.config.ts
|-- wdio.conf.js
|-- package.json
|-- requirements.txt
|-- Dockerfile
|-- docker-compose.yml
`-- README.md

Prerequisites

Installation

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

OPENAI_API_KEY=your_key

$env:OPENAI_API_KEY = "your_key"

Optional: GitAgent (Repo-Specific)

This repository includes a targeted gitagent setup for its QA automation workflow:

• agent.yaml (manifest)
• SOUL.md and RULES.md (behavior and constraints)
• knowledge/ (framework and repo references)

In short: agent.yaml defines the repo agent, SOUL.md and RULES.md define how it should behave, and knowledge/ gives it project-specific framework guidance.

Quick commands:

npm run gitagent:validate
npm run gitagent:info
npm run gitagent:export

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
docker compose build

docker compose run --rm test-generator "Test login" --url https://the-internet.herokuapp.com/login

docker compose run --rm test-generator \
  "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

GitHub Registry (GHCR)

Pre-built Docker images are published to GitHub Container Registry. No local clone or build required.

Pull and run

docker pull ghcr.io/aiqualitylab/ai-natural-language-tests:latest

docker run --rm \
  -e OPENAI_API_KEY=your_key \
  ghcr.io/aiqualitylab/ai-natural-language-tests:latest \
  "Test login" --url https://the-internet.herokuapp.com/login

Image tags

For publishing and release management, see CONTRIBUTING.md.

Configuration

OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
GOOGLE_API_KEY=your_key

OTEL_PROVIDER=grafana
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-north-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instance_id:api_token)>

GRAFANA_LOKI_URL=https://logs-prod-eu-north-0.grafana.net
GRAFANA_INSTANCE_ID=<instance_id>
GRAFANA_API_TOKEN=<logs_write_token>

[!TIP] Privacy-first setup:

• Use an LLM provider account/plan that guarantees no training or zero-retention for API data.
• Keep OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GRAFANA_LOKI_URL, GRAFANA_INSTANCE_ID, and GRAFANA_API_TOKEN unset to avoid telemetry/log shipping.
• Use masked or synthetic test data for sensitive fields.

[!TIP] Need to support a new URL and tune prompts safely? Follow the step-by-step guide in PROMPT_UPDATE_GUIDE.md.

Usage

Quick Reference

[!TIP] If your global python misses project dependencies, run with the repository virtual environment:

• PowerShell: .\.venv\Scripts\python.exe qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run
• Bash: ./.venv/Scripts/python.exe qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

[!NOTE] The current CLI supports URL-driven generation via --url. A direct --data JSON input flag is not implemented in this repository yet.

Natural Language Prompt Examples

[!TIP] Writing effective AI requirements

• Be specific about the action: "Test login" vs "Test login with valid credentials and verify dashboard loads"
• Mention the expected outcome when it matters: "...and verify error message appears"
• Use --url to give the AI real page context — it reads the HTML and picks the right selectors automatically
• Chain multiple requirements in one run: "Test login" "Test logout" --url <url>

Generate Cypress Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login

Generate Playwright Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright

Generate WebdriverIO Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework webdriverio

Prompt-Powered Cypress Mode

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --use-prompt

Generate and Execute

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

Failure Analysis

python qa_automation.py --analyze "CypressError: Element not found"
python qa_automation.py --analyze -f error.log

[!NOTE] The AI failure analyzer returns a structured diagnosis:

Field	Description
CATEGORY	Error type: SELECTOR, TIMEOUT, ASSERTION, NETWORK, etc.
REASON	Root cause explanation in plain English
FIX	Suggested code change or configuration fix

Pattern Inventory

python qa_automation.py --list-patterns

CI/CD Integration

flowchart TD
    A[Code Changes<br/>Pushed to Repo] --> B[CI/CD Pipeline<br/>Triggers]
    B --> C[Install Dependencies<br/>pip install -r requirements.txt<br/>npm install]
    C --> D[Generate Tests<br/>python qa_automation.py<br/>--url]
    D --> E[Run Tests<br/>npx cypress run<br/>npx playwright test<br/>npx wdio run]
    E --> F{Tests Pass?}
    F -->|Yes| G[Deploy Application<br/>Success]
    F -->|No| H[AI Failure Analysis<br/>--analyze in pipeline]
    H --> I[Auto-Fix & Regenerate<br/>If possible]
    I --> E
    H --> J[Notify Developers<br/>Manual intervention]

    style A fill:#e1f5fe,color:#333333,stroke:#666666
    style B fill:#fff3e0,color:#333333,stroke:#666666
    style C fill:#c8e6c9,color:#333333,stroke:#666666
    style D fill:#ffcdd2,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style G fill:#e8f5e8,color:#333333,stroke:#666666
    style J fill:#ffebee,color:#333333,stroke:#666666

Recommended pipeline stages:

Security and Compliance Guidance

[!IMPORTANT]

• Store secrets only in secure secret managers (never commit .env).
• Use scoped API tokens with least-privilege access.
• Rotate provider keys and Grafana tokens on a fixed cadence.
• Keep generated tests and reports free of sensitive production data.
• Apply repository protection rules and mandatory CI checks.

Troubleshooting

[!WARNING] Traces Not Visible in Grafana Tempo

• Verify OTLP endpoint region and datasource selection.
• Verify Authorization=Basic <base64(instance_id:api_token)> format.
• Query with:

{resource.service.name="ai-natural-language-tests"}

[!NOTE] Loki Authentication Errors

• Ensure token has logs:write scope.
• Confirm instance ID and logs endpoint match the same Grafana stack.

[!TIP] Docker Observability Validation

• Confirm .env includes OTLP and Loki keys before docker compose run.
• Use docker compose config to verify environment interpolation.
• In Grafana Explore, query Tempo with service.name="ai-natural-language-tests".
• In Grafana Loki, query labels: {service_name="ai-natural-language-tests"}.

[!TIP] Switching to Headed Mode for Debugging

Tests run headless by default. To debug interactively, switch your framework config:

Cypress:

• Edit cypress.config.js and add headed: true after browser: 'chrome'
• Or run: npx cypress run --headed --spec 'cypress/e2e/generated/*.cy.js'

Playwright:

• Edit playwright.config.ts and change headless: true → headless: false
• Or run: npx playwright test --headed tests/generated/

WebdriverIO:

• Edit wdio.conf.js and comment out '--headless=new' from the args array

Docker Headed Mode (with X11 forwarding):

docker build --target debug -t ai-tests:debug .
docker run -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix ai-tests:debug

• Optional: mainly for Linux visual debugging.

• Retry with generated single-spec command from logs.

Documentation Map

Versioning and Release Policy

Support and Security Reporting

Compliance and Data Handling

For implementation details and contribution controls, see CONTRIBUTING.md.

Operational Expectations

These are practical runbook-style expectations for delivery teams. They are operational targets, not contractual SLAs.

Support Matrix

The matrix below reflects currently configured and documented project baselines.

Changelog

Release notes are maintained in CHANGELOG.md, following the Keep a Changelog format.

Production-focused AI-assisted E2E test generation for modern QA teams.

© 2026 AI Quality Lab / Sreekanth Harigovindan.

tests.aiqualitylab.org