Custom end-to-end API contract tests with Arazzo

You can use Speakeasy to create custom end-to-end contract tests that run against a real API.

This document guides you through writing more complex tests using the Arazzo Specification, as well as through the key configuration features for these tests, which include:

• Server URLs
• Security credentials
• Environment variable-provided values

Arazzo is a simple, human-readable, and extensible specification for defining API workflows. Arazzo powers custom test generation, allowing you to define rich tests capable of:

• Testing multiple operations
• Testing different inputs
• Validating that the correct response is returned
• Running against a real API or mock server
• Configuring setup and teardown routines for complex end-to-end (E2E) tests

The Arazzo Specification allows you to define sequences of API operations and their dependencies for contract testing, enabling you to validate whether your API behaves correctly across multiple interconnected endpoints and complex workflows.

When a .speakeasy/tests.arazzo.yaml file is found in your SDK repo, the Arazzo workflow is used to generate tests for each of the workflows defined in the file.

Prerequisites

Before generating tests, ensure that you meet the testing feature prerequisites.

Writing custom end-to-end tests

The following is an example Arazzo document defining a simple E2E test for the lifecycle of a user resource in the example API:

This workflow defines four steps, each of which feeds into the next:

1. Create a user
2. Retrieve that user via its new ID
3. Update the user
4. Delete the user

This is possible because the workflow defines outputs for certain steps that serve as inputs for the following steps.

The workflow generates the test shown below:

Input and outputs

Inputs

There are various ways to provide an input for a step, including:

• Defining it in the workflow
• Referencing it in a previous step
• Including it as an inline value

Workflow inputs

You can provide input parameters to the workflow using the inputs field, which is a JSON Schema object that defines a property for each input that the workflow wants to expose. These workflow inputs can be used by any step defined in the workflow.

Test generation can use any of the examples defined for a property in an inputs JSON schema as a literal value, which it then uses as an input for the test. Because tests are non-interactive and cannot ask users for input, the test generation randomly generates values for the inputs if no examples are defined.

Step references

Parameters and request body payloads can reference values via runtime expressions from previous steps in the workflow. This allows for the generation of tests that are more complex than a simple sequence of operations.

Speakeasy’s implementation currently only allows the referencing of a previous step’s output, which means you will need to define which values you want to expose to future steps.

Inline values

For any parameters or request body payloads that a step defines, you can also provide literal values inline to populate the tests (if static values are suitable for the tests).

Payload values

If you use the payload field of a request body input, its value can be any of the following:

• A static value
• A value with interpolated runtime expressions
• A runtime expression by itself

You can then overlay the payload value using the replacements field, which represents a list of targets within the payload that will be replaced with the value of the replacements. These replacements can be static values or runtime expressions.

Outputs

As shown above, you can define outputs for each of the steps in a workflow, allowing you to use values from things such as response bodies in following steps.

Currently, Speakeasy supports only referencing values from a response body, using the runtime expressions syntax and json-pointers.

Any number of outputs can be defined for a step.

Success criteria

A step’s successCriteria field contains a list of criterion objects used to validate the success of the step and form the basis of the test assertions for test generation.

The successCriteria may be as simple as a single condition testing the status code of the response or as complex as multiple conditions testing various individual fields within the response body.

Speakeasy’s implementation currently only supports simple criteria and the use of the equality (==) and inequality (!=) operators for comparing values and for testing status codes, response headers, and response bodies.

Note: While the Arazzo specification defines additional operators like >, <, >=, <=, ~, and !~, Speakeasy currently only supports == and !=.

To test values within the response body, due to the typed nature of the SDKs, you need criteria for testing the status code and content type of the response to help the generator determine which response schema to validate against.

Testing operations requiring binary data

Some operations require you to provide binary data, for example, for testing file uploads and downloads.

To provide the test with the test files, use the x-file directive in the example for the relevant field.

The files are sourced from the .speakeasy/testfiles directory in the root of your SDK repo, where the path provided in the x-file directive is relative to the testfiles directory.

The content of the sourced file is used as the value for the field being tested.

Configuring an API to test against

By default, tests are generated to run against Speakeasy’s mock server, which has a URL of http://localhost:18080. The mock server validates that the SDKs are functioning correctly but does not guarantee that the API is correct.

The generator can be configured to run all tests against another URL or against individual tests using the x-speakeasy-test-server extensions in the .speakeasy/tests.arazzo.yaml file.

If the extension is found at the top level of the Arazzo file, all workflows and tests will be configured to run against the specified server URL. If the extension is found within a workflow, only that workflow will be configured to run against the specified server URL.

The server URL can be either a static URL or an x-env: EXAMPLE_ENV_VAR value that pulls the value from the environment variable, EXAMPLE_ENV_VAR (where the name of the environment variable can be any specified name).

You may provide a default value in the x-env directive if the environment variable is not set. This can be useful for local development or non-production environments.

If all tests are configured to run against other server URLs, you can disable mock server generation in the .speakeasy/gen.yaml file:

Configuring security credentials for contract tests

When running tests against a real API, the SDK may need to be configured with security credentials to authenticate with the API. To configure the SDK, add the x-speakeasy-test-security extension to the document, workflow, or individual step.

The x-speakeasy-test-security extension allows static values, values pulled from the environment, or runtime expressions referencing outputs from previous steps to be used when instantiating an SDK instance and making requests to the API.

Important: The keys under value must exactly match the names of the securitySchemes defined in your OpenAPI document’s components.securitySchemes section.

For example, if your OpenAPI document defines:

Then your Arazzo test configuration should reference these exact scheme names:

Using Runtime Expressions for Dynamic Security

The x-speakeasy-test-security extension also supports runtime expressions, allowing you to populate security credentials dynamically from the outputs of previous steps. This is particularly useful for workflows that require authentication tokens obtained from login operations.

For example, you can use a runtime expression to reference a token from a previous authentication step:

In this example:

• The workflow defines x-speakeasy-test-security at the workflow level with apiKey: $steps.authenticate.outputs.token
• The first step (authenticate) calls an authentication workflow and captures the token in its outputs
• All subsequent steps in the workflow will use this dynamically obtained token for authentication
• The runtime expression $steps.authenticate.outputs.token references the token output from the authenticate step

This approach enables complex authentication flows where tokens must be obtained dynamically during test execution, rather than being provided as static values or environment variables.

Configuring environment variable provided values for Contract tests

When running tests against a real API, you may need to fill in certain input values from dynamic environment variables. Use the Speakeasy environment variable extension to do so: