Acceptance tests

Tip

Find more information about acceptance tests in this section of the Nextmv docs.

Define the desired acceptance test ID and name. As mentioned above, an acceptance test is based on a batch experiment.

• If you already started a batch experiment, you don't need to provide the input_set_id parameter. In that case, the ID and name of the acceptance test and the underlying batch experiment must be the same.
• If you didn't start a batch experiment, you need to provide the input_set_id and a new batch experiment will be created for you, with the same ID and name as the acceptance test.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application(client=client, id="<YOUR_APP_ID>")
acceptance_test = app.new_acceptance_test(
    candidate_instance_id="latest-2",
    control_instance_id="latest",
    id="<YOUR_ACCEPTANCE_TEST_ID>",
    name="<YOUR_ACCEPTANCE_TEST_ID>",
    metrics=[
        cloud.Metric(
            field="result.value",
            metric_type=cloud.MetricType.direct_comparison,
            params=cloud.MetricParams(operator=cloud.Comparison.less_than),
            statistic="mean",
        ),
        cloud.Metric(
            field="result.custom.activated_vehicles",
            metric_type=cloud.MetricType.direct_comparison,
            params=cloud.MetricParams(operator=cloud.Comparison.greater_than),
            statistic="mean",
        ),
    ],
    # input_set_id=os.getenv("INPUT_SET_ID"), # Defining this would create a new batch experiment.
    description="An optional description",
)
nextmv.write(acceptance_test)

$ python main.py
{
  "id": "experiment-1",
  "name": "experiment-1",
  "description": "An optional description",
  "app_id": "routing",
  "experiment_id": "experiment-1",
  "control": {
    "instance_id": "latest",
    "version_id": "v1.0.4"
  },
  "candidate": {
    "instance_id": "latest-2",
    "version_id": "v1.1.0"
  },
  "metrics": [
    {
      "field": "result.value",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "lt"
      },
      "statistic": "mean"
    },
    {
      "field": "result.custom.activated_vehicles",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "gt"
      },
      "statistic": "mean"
    }
  ],
  "created_at": "2024-01-12T19:30:29.677473Z",
  "updated_at": "2024-01-12T19:30:29.677473Z"
}

Delete an acceptance test

Deleting an acceptance test will also delete all of the associated information. Note that deleting it will not delete the underlying batch experiment and its runs.

Warning

This action is permanent and cannot be undone.

You can delete an acceptance test with the Application.delete_acceptance_test method.

import os

from nextmv.cloud import Application, Client

client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = Application(client=client, id="<YOUR_APP_ID>")
app.delete_acceptance_test(acceptance_test_id="<YOUR_ACCEPTANCE_TEST_ID>")

You will not be prompted to confirm the deletion.