Run a prompt's test cases

POST/v1/prompts/{slug}/test-cases/run

Run a prompt's test cases.

Renders the prompt's default version (the production label, else the PUBLISHED version, else the highest version) with each test case's variables and sends it to the model on the given virtual key, one test case after another. A test case passes when its answer matches its expected_pattern (a regular expression, any case; plain text when it is not a valid one), or has no pattern. Each call is billed to the virtual key. A test case whose call fails is reported as failed with its error; the run itself still succeeds. Requires an admin of the prompt's workspace, or an organization admin.

Errors: 400 when the virtual key is unknown or not usable in the prompt's workspace, or the version has no model and no model_override is given; 404 when the prompt has no versions.

Authorization

AuthorizationBearer va_
A platform API key (va_...), created under Settings > API keys. Scoped to an organization or one workspace, with an admin or member role. Used by the management API.

Path parameters

slugstringrequired
the prompt's slug

Query parameters

organizationstringrequired
slug of the organization that owns it

Request body application/json

the virtual key, and an optional model override

virtual_keystringrequired
Slug of the virtual key that runs the test cases (required), for example vk-.... It must belong to the prompt's organization and, when it is scoped to a workspace, to the prompt's workspace (400 otherwise). Each test case is one completion on this key, with its usage and cost.
model_overridestring
A model to use instead of the version's model. Required when the version has no model (400 otherwise).

Responses

200application/json

prompt_slugstring
Slug of the prompt.
version_numberinteger<int32>
The version that was tested: the one the production label points at, else the PUBLISHED version, else the highest version number.
virtual_keystring
Slug of the virtual key the test cases ran on.
test_case_countinteger<int32>
Number of test cases run.
pass_countinteger<int32>
Number of test cases that passed.
fail_countinteger<int32>
Number of test cases that failed, including calls that errored.
total_cost_usdnumber<double>
Total cost of the run in USD, the sum of the test cases' cost_usd. Null when the prompt has no test cases.
resultsobject[]
One result per test case, in the order they ran.
Child attributes
- test_case_idstring
  Id (UUID) of the test case.
- namestring
  The test case's name.
- passedboolean
  Whether the answer matched the test case's expected_pattern. false when the call failed.
- response_textstring
  The model's answer text. Null when the call failed.
- latency_msinteger<int64>
  How long the call took, in milliseconds. Null when it failed.
- prompt_tokensinteger<int32>
  Input tokens the call used. Null when it failed or the provider reported no usage.
- completion_tokensinteger<int32>
  Output tokens the call generated. Null when it failed or the provider reported no usage.
- cost_usdnumber<double>
  The provider cost of the call in USD, at list price. Null when the call failed or the model has no price.
- error_messagestring
  Why the call failed. Null when it succeeded.
- pattern_matchedboolean
  Whether the answer matched expected_pattern (true when the test case has no pattern and the call succeeded). Same as passed.

Request

curl -X POST "https://api.vectoraxis.ai/v1/prompts/{slug}/test-cases/run?organization=string" \
  -H "Authorization: Bearer $VECTORAXIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "virtual_key": "string"
}'

VectorAxis is built by GK Labs, a member of NVIDIA Inception.