Logo
Testing Platform

Testing Platform

Ship AI agents with confidence. Define test cases with expected inputs and outputs, run automated suites from the CLI or your CI/CD pipeline, and track results over time with full deployment traceability.

Core Concepts

Test Cases

Define individual test scenarios with input messages and expected response criteria. Group related cases into test suites.

Test Environments

Configure isolated environments with specific agent versions and settings for consistent, reproducible testing.

Test Runs

Execute test suites against specific deployments. View pass/fail results, response times, and detailed output comparisons.

Deployments

Track agent deployments across environments. Link test runs to specific deployment versions for traceability.

CLI in Action

The smooai-testing CLI lets you manage and run tests directly from your terminal or CI/CD pipeline.

smooai-testing
$ smooai-testing login
Authenticated as [email protected]
$ smooai-testing cases list
ID Name Status
case_abc123 Greeting Test active
case_def456 Order Lookup active
case_ghi789 Escalation Flow active
case_jkl012 Edge Case: Empty Input active
$ smooai-testing runs create --env staging --deployment v2.1.0
Creating test run... done
Run ID: run_abc123
Running 4 test cases...
✓ Greeting Test passed (1.2s)
✓ Order Lookup passed (2.1s)
✓ Escalation Flow passed (1.8s)
✗ Edge Case: Empty Input failed (0.4s)
Results: 3/4 passed, 1 failed
pnpm add -D @smooai/testing-cli

Multi-Language Support: The Testing CLI and SDK are currently TypeScript-only. For other languages, you can use the REST API to integrate test reporting from any language or framework that supports CTRF.

CTRF Integration

Smoo Testing natively supports the Common Test Report Format (CTRF). Test results from any CTRF-compatible runner automatically flow into the Smoo AI dashboard with zero configuration.

import { SmooTestingClient } from '@smooai/testing';

const client = new SmooTestingClient({
    token: process.env.SMOO_ACCESS_TOKEN!,
});

// Upload a CTRF report from any test runner
await client.uploadCtrfReport({
    environmentId: 'env_staging',
    deploymentId: 'deploy_v2.1.0',
    report: './ctrf-report.json',
});

Supported Test Runners

Any runner with a CTRF reporter: Playwright, Jest, Vitest, Mocha, pytest, JUnit, and more. Results are automatically parsed and displayed in the Smoo AI dashboard.

GitHub Integration

Connect your GitHub repository for one-click sync of deployments, environments, and test results. Every push creates a deployment record, and test runs link back to the commit that triggered them.

Auto Deployments

Pushes to configured branches automatically create deployment records with commit metadata.

PR Status Checks

Test results post directly to pull requests as GitHub status checks with pass/fail indicators.

Actions Integration

Drop-in GitHub Action for running tests on every push, PR, or scheduled interval.

SmooTestingClient SDK

The TypeScript SDK provides programmatic access to the full testing API. Create test cases, trigger runs, and fetch results from your application code.

import { SmooTestingClient } from '@smooai/testing';

const testing = new SmooTestingClient({
    token: process.env.SMOO_ACCESS_TOKEN!,
    organizationId: 'org_abc123',
});

// Create a test case
const testCase = await testing.createTestCase({
    name: 'Order Status Inquiry',
    input: 'What is the status of order #12345?',
    expectedOutput: 'Includes order status and estimated delivery date',
    agentId: 'agent_xyz',
});

// Start a test run
const run = await testing.createTestRun({
    environmentId: 'env_staging',
    deploymentId: 'deploy_v2.1.0',
    testCaseIds: [testCase.id],
});

// Poll for results
const results = await testing.getTestRun(run.id);
console.log(results.summary);
// { total: 1, passed: 1, failed: 0, duration: '1.2s' }

CI/CD Pipeline

Run tests automatically on every deployment. Here's a GitHub Actions workflow that runs your test suite after deploying to staging.

name: Test AI Agent
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to staging
        run: pnpm deploy:staging

      - name: Run Smoo AI tests
        env:
          SMOO_ACCESS_TOKEN: ${{ secrets.SMOO_ACCESS_TOKEN }}
        run: |
          npx smooai-testing runs create \
            --env staging \
            --deployment ${{ github.sha }} \
            --wait \
            --fail-on-error

CLI Flags

  • --wait — Block until the test run completes
  • --fail-on-error — Exit with non-zero code if any test fails
  • --format ctrf — Output results in CTRF format for downstream processing

REST API Examples

Create a Test Case

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-cases \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Greeting Test",
    "description": "Verify the agent responds with a friendly greeting",
    "input": "Hello, I need help with my order",
    "expectedOutput": "Contains a greeting and asks for order details",
    "agentId": "agent_id"
  }'

Create a Test Environment

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-environments \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Staging",
    "description": "Pre-production testing environment"
  }'

Start a Test Run

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "testEnvironmentId": "env_id",
    "deploymentId": "deployment_id",
    "testCaseIds": ["case_1", "case_2", "case_3"]
  }'

Get Test Run Results

curl https://api.smoo.ai/organizations/{org_id}/test-runs/{run_id} \
  -H "Authorization: Bearer YOUR_TOKEN"

Typical Workflow

  1. 1

    Define test cases

    Create test cases that cover your agent's expected behavior: greetings, FAQs, edge cases, and escalation scenarios.

  2. 2

    Set up environments

    Create test environments that mirror your deployment stages (development, staging, production).

  3. 3

    Run tests on each deployment

    Execute test suites after each deployment to catch regressions. Use the CLI or GitHub Actions for automation.

  4. 4

    Review results and iterate

    Analyze test run results in the dashboard, identify failing cases, and refine your agent's knowledge base and prompts.