Test with environments and saved cases

If you've used Postman, the console's test tools will feel familiar:

Environments, test cases and runs live in the console's developer pane, under Test data (press Ctrl + ` on any project page, or open Developer at the bottom of the sidebar). The endpoint editor keeps a quick Test panel for trying inputs and saving them as cases.

Everything here runs through the same console test route as the Test panel: test runs are never cached, cost credits like any model call and show up in usage with kind test.

Environments

An environment is a named set of variables for test inputs: sample customers, order IDs, a long ticket you don't want to paste again. Environments belong to the project, so every collection in it can use them. Manage them in the developer pane under Test data → Environments (a link with ?dev=test-data opens it), or with the gear next to the environment picker in an endpoint's Test panel, which opens the same view.

Each variable has a key (a letter or _, then letters, digits or _), a value, an optional description and a Secret checkbox. A value is text or JSON: switch with Text / JSON, or start typing { or [ and it switches to JSON for you.

{
  "customer_plan": "pro",
  "sample_ticket": { "subject": "Can't log in", "body": "The login page spins forever." }
}

Secret values are masked once saved: the console shows Stored value hidden and only fetches the value when you choose Reveal. Saving the environment without revealing keeps the stored value. A project has up to 50 environments with up to 200 variables each.

Using variables in inputs

Write {{env.key}} anywhere in the test inputs and pick an environment at the top of the Test panel (the choice is remembered per project in this browser):

{
  "ticket": "{{env.sample_ticket}}",
  "customer": { "plan": "{{env.customer_plan}}" },
  "language": "Reply in {{env.language}}, please"
}

• An input that is exactly {{env.key}} gets the variable's value with its type: above, ticket becomes the whole object.
• Inside longer text, the value is inserted as text (objects and numbers as compact JSON).
• Placeholders work at any depth, in objects and arrays.

Below the inputs the panel shows Resolves with …, the inputs as they will be sent, and warns about keys the environment doesn't have (or secrets without a value). A run with missing keys doesn't start.

Saved test cases

Save as test case in the Test panel stores the current inputs, placeholders and all, with the selected environment and a name. Add checks right away or later. Once it's saved, the developer pane opens on Test data → Test cases, which lists the endpoint's cases (the endpoint on screen, or pick a collection and an endpoint) with the last result of each (kept while the browser tab is open):

• Run runs one case against the editor's current draft (or the saved definition when the endpoint isn't open), with the case's environment.
• Load into Test puts its inputs and environment into the endpoint editor's Test panel, opening the editor if you're on another page.
• Edit changes the name, the inputs (JSON), the environment and the checks; Duplicate, Delete, Move up / down do what they say.

An endpoint keeps up to 200 cases, each with inputs up to 256 KB.

Checks

A case passes when every check passes (a case without checks passes when the run succeeds). Paths are dot paths into the run's response: data.category, data.tags.0, model, cached, usage.cost_usd. The path field suggests paths from the endpoint's output schema.

Patterns are JavaScript regular expressions, up to 500 characters, without lookbehind, backreferences, (?…) groups other than (?:, (?= and (?!, \A/\Z, a ] first in a class, or nested unbounded repeats such as (a+)+.

Add checks from this result (or Add suggested checks before a run) adds the obvious ones: status ok, required fields exist, and enum fields stay within their allowed values.

Running all cases

Test data → Runs in the developer pane runs saved cases: choose Cases to run (the whole collection, or one endpoint). Run all in Test cases opens it for that endpoint, and Run all test cases on the collection's Endpoints tab opens it for the whole collection. Choose:

• Run against: the Draft, or a published version such as v1 (the version's frozen definition runs; cases for endpoints the version doesn't have are skipped).
• Environment: Each case's own, no environment, or one environment for every case (to run the same cases against staging and production data, for example).

Cases run two at a time with a progress bar, and you can cancel. The results table shows each case's result, the failed checks with expected and actual values, latency, cost and model; open a row for its output. The summary counts passed, failed and errored cases with the total cost. Run again repeats the run, Run failed only reruns the failed and errored cases. Results stay until you close the browser tab, and a run keeps going while you switch to another tab of the pane.

Deleting an environment doesn't delete the cases that used it: they run with no environment afterwards.