Testing

End-to-end API testing with Arazzo, TypeScript, and Deno

Brian Flad

October 30, 2024 - 17 min read

Testing

---

We’ve previously written about the importance of building contract & integration tests to comprehensively cover your API’s endpoints, but there’s still a missing piece to the puzzle. Real users don’t consume your API one endpoint at a time - they implement complex workflows that chain multiple API calls together.

That’s why reliable end-to-end API tests are an important component of the testing puzzle. For your APIs most common workflows, you need to ensure that the entire process works as expected, not just the individual parts.

In this tutorial, we’ll build a test generator that turns Arazzo specifications into executable end-to-end tests. You’ll learn how to:

• Generate tests that mirror real user interactions with your API
• Keep tests maintainable, even as your API evolves
• Validate complex workflows across multiple API calls
• Catch integration issues before they reach production

We’ll use a simple “Build-a-bot” API as our example, but the principles and code you’ll learn apply to any REST API.

Arazzo? What & Why

Arazzo is a specification that describes how API calls should be sequenced to achieve specific outcomes. Think of it as OpenAPI for workflows - while OpenAPI describes what your API can do, Arazzo describes how to use it effectively.

Arazzo was designed to bridge the gap between API reference documentation and real-world usage patterns. Fortunately for us, it also makes a perfect fit for generating end-to-end test suites that validate complete user workflows rather than isolated endpoints.

By combining these specifications, we can generate tests that validate not just the correctness of individual endpoints, but the entire user journey.

Let’s look at a simplified (and mostly invalid) illustrative example. Imagine a typical e-commerce API workflow:

Arazzo allows us to define these workflows, and specify how each step should handle success and failure conditions, as well as how to pass data between steps and even between workflows.

From specification to implementation

The example above illustrates the concept, but let’s dive into a working implementation. We’ll use a simplified but functional example that you can download and run yourself. Our demo implements a subset of the Arazzo specification, focusing on the most immediately valuable features for E2E testing.

We’ll use the example of an API called Build-a-bot, which allows users to create and manage their own robots. You can substitute this with your own OpenAPI document, or use the Build-a-bot API to follow along.

If you want a deep dive on the Arazzo specification, check out the Arazzo specification docs.

Setting up the development environment

First, clone the demo repository:

You’ll need Deno v2  installed. On macOS and Linux, you can install Deno using the following command:

The repository contains:

• A simple API server built with @oak/acorn  that serves as the Build-a-bot API (in packages/server/server.ts)
• An Arazzo specification file (arazzo.yaml)
• An OpenAPI specification file (openapi.yaml)
• The test generator implementation (packages/arazzo-test-gen/generator.ts)
• Generated E2E tests (tests/generated.test.ts)
• An SDK created by Speakeasy to interact with the Build-a-bot API (packages/sdk)

Running the Demo

To run the demo, start the API server:

Deno will install the server’s dependencies, then start the server on http://localhost:8080. You can test the server by visiting http://localhost:8080/v1/robots, which should return a 401 Unauthorized error:

Next, in a new terminal window, generate the E2E tests:

After installing dependencies, this command will generate the E2E tests in tests/generated.test.ts and watch for changes to the Arazzo specification file.

You can run the tests in a new terminal window:

This command will run the generated tests against the API server:

Beautiful, everything works! Let’s see how we got here.

Building an Arazzo test generator

Let’s start with the overall structure of the test generator.

Project structure

The test generator is a Deno project that consists of several modules, each with a specific responsibility:

• generator.ts: The main entry point that orchestrates the test generation process. It reads the Arazzo and OpenAPI specifications, validates their compatibility, and generates test cases.

• readArazzoYaml.ts and readOpenApiYaml.ts: Handle parsing and validation of the Arazzo and OpenAPI specifications respectively. They ensure the specifications are well-formed and contain all required fields.

• expressionParser.ts: A parser for runtime expressions like $inputs.BUILD_A_BOT_API_KEY and $steps.createRobot.outputs.robotId. These expressions are crucial for passing data between steps and accessing workflow inputs.

• successCriteria.ts: Processes the success criteria for each step, including status code validation, regex patterns, direct comparisons, and JSONPath expressions.

• generateTestCase.ts: Takes the parsed workflow and generates the actual test code, including setup, execution, and validation for each step.

• security.ts: Handles security-related aspects like API key authentication and other security schemes defined in the OpenAPI specification.

• utils.ts: Contains utility functions for common operations like JSON pointer resolution and type checking.

The project also includes a runtime-expression directory containing the grammar definition for runtime expressions:

• runtimeExpression.peggy: A Peggy grammar file that defines the syntax for runtime expressions
• runtimeExpression.js: The generated parser from the grammar
• runtimeExpression.d.ts: TypeScript type definitions for the parser

Let’s dive deeper into each of these components to understand how they work together to generate effective E2E tests.

Parsing until you parse out

While our project says “test generator” on the tin, the bulk of our work will go into parsing different formats. To generate tests from an Arazzo document, we need to parse:

1. The Arazzo document
2. The OpenAPI document
3. Conditions in the Arazzo success criteria
4. Runtime expressions in the success criteria, outputs, and parameters
5. Regular expressions in the success criteria
6. JSONPath expressions in the success criteria
7. JSON pointers in the runtime expressions

We won’t cover all of these in detail, but we’ll touch on each to get a sense of the complexity involved and the tools we use to manage it.

Parsing the Arazzo specification

The first step in our test generator is parsing the Arazzo specification in readArazzoYaml.ts. This module reads the Arazzo YAML file and should ideally validate its structure against the Arazzo specification.

For our demo, we didn’t implement full validation, instead parsing the YAML file into a JavaScript object. We then use TypeScript interfaces to define the expected structure of the Arazzo document:

These TypeScript interfaces help with autocompletion, type checking, and documentation, making it easier to work with the parsed Arazzo document in the rest of our code.

The real complexity comes in validating that the parsed document follows all the rules in the Arazzo specification. For example:

• Each workflowId must be unique within the document
• Each stepId must be unique within its workflow
• An operationId must reference a valid operation in the OpenAPI document
• Runtime expressions must follow the correct syntax
• Success criteria must use valid JSONPath or regex patterns

We don’t validate all these rules in our demo, but in production, we’d use Zod  or Valibot  to enforce these constraints at runtime and provide helpful error messages when the document is invalid.

The OpenAPI team hasn’t finalized the Arazzo specification’s JSON Schema yet, but once they do, we can use it to validate the Arazzo document against the schema with tools like Ajv .

Speakeasy also provides a command-line interface for linting Arazzo documents:

Parsing the OpenAPI specification

The OpenAPI specification’s path is gathered from the Arazzo document. In our test, we simply use the first sourceDescription to find the OpenAPI document path. But in a production generator, we’d need to handle multiple sourceDescriptions and ensure the OpenAPI document is accessible.

We parse the OpenAPI document in readOpenApiYaml.ts and use TypeScript interfaces from the npm:openapi-types package to define the expected structure of the OpenAPI document.

We won’t cover the OpenAPI parsing in detail, but it’s similar to the Arazzo parsing: Read the YAML file, parse it into a JavaScript object, and type check it against TypeScript interfaces.

For OpenAPI, writing a custom validator is more complex due to the specification’s size and complexity. We recommend validating against the official OpenAPI 3.1.1 JSON Schema  using Ajv , or Speakeasy’s own OpenAPI linter:

Parsing success criteria

This is where things get interesting. Success criteria in Arazzo are a list of conditions that must be met for a step to be considered successful. Each criterion can be one of the following types:

• simple: Selects a value with a runtime expression and compares it to an expected value
• jsonpath: Selects a value using a JSONPath expression and compares it to an expected value
• regex: Validates a value against a regular expression pattern
• xpath: Selects a value using an XPath expression and compares it to an expected value, used for XML documents

In our demo, we don’t implement the xpath type, but we do cover the other three. Here’s an example of a success criterion in the Arazzo document:

The condition field is required, and contains the expression to evaluate, while the context field specifies the part of the response to evaluate. The type field indicates the type of validation to perform.

If no type is specified, the success criterion is treated as a simple comparison, where the condition is evaluated directly.

Evaluating simple criteria

Here’s an example of how we parse a simple success criterion:

We split this simple condition into:

• Left-hand side: $statusCode - Runtime expression to evaluate
• Operator: == - Comparison operator or assertion
• Right-hand side: 201 - Expected value

We’ll evaluate the runtime expression $statusCode and compare it to the expected value 201. If the comparison is true, the criterion passes; otherwise, it fails.

Runtime expressions can also reference other variables, like $inputs.BUILD_A_BOT_API_KEY or $steps.createRobot.outputs.robotId, or fields in the response body, like $response.body#/model.

We’ll cover runtime expressions in more detail later.

Evaluating JSONPath criteria

For JSONPath criteria, we use the jsonpath type and a JSONPath expression to select a value from the response:

Let’s break down the JSONPath criterion:

• context: $response.body#/links - Runtime expression to select the links array from the response body
• condition: $.length == 5 - JSONPath expression compared to an expected value
• type: jsonpath - Indicates the criterion type

We further need to break down the condition into:

• Left-hand side: $.length - JSONPath expression to evaluate
• Operator: == - Comparison operator
• Right-hand side: 5 - Expected value

We evaluate the JSONPath expression $.length and compare it to the expected value 5. If the comparison is true, the criterion passes.

Evaluating regex criteria

For regex criteria, we use the regex type and a regular expression pattern to validate a value:

Let’s break down the regex criterion:

• context: $response.body#/robotId - Runtime expression to select the robotId field from the response body
• condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i - Regular expression pattern to validate the value as a UUID v4
• type: regex - Indicates the criterion type

We evaluate the runtime expression $response.body#/robotId against the regular expression pattern. If the value matches the pattern, the criterion passes.

In our implementation, we use TypeScript’s factory.createRegularExpressionLiteral to create a regular expression literal from the pattern string. This ensures that the pattern is properly escaped and formatted as a valid JavaScript regular expression.

The generated test code looks something like this:

This code uses Deno’s built-in assertMatch function to validate that the robotId matches the UUID v4 pattern. If the value doesn’t match, the test fails with a helpful error message.

Parsing runtime expressions

Runtime expressions are used throughout the Arazzo specification to reference variables, fields in the response body, or outputs from previous steps. They follow a specific syntax defined in the Arazzo specification as an ABNF (augmented Backus–Naur form) grammar.

To parse runtime expressions, we use a parser generated from the ABNF grammar. In our demo, this is a two-step process. First, we use the abnf  npm package to generate a Peggy grammar file from the ABNF grammar:

This generates a runtime-expression/runtimeExpression.peggy file that defines the syntax for runtime expressions. We then use the peggy  npm package to generate a parser from the Peggy grammar:

This generates a runtime-expression/runtimeExpression.js file that contains the parser for runtime expressions. We also generate TypeScript type definitions in runtime-expression/runtimeExpression.d.ts.

The parser reads a runtime expression like $response.body#/robotId and breaks it down into tokens. We then evaluate the tokens to resolve the expression at runtime.

Evaluating runtime expressions

Once we’ve parsed a runtime expression, we need to evaluate it to get the value it references. For example, given the expression $response.body#/robotId, we need to extract the robotId field from the response body.

The evaluateRuntimeExpression function in utils.ts handles this evaluation. Here’s an example of how it works:

Here, we handle two types of runtime expressions: $statusCode and $response.body. We extract the status field from the response object for $statusCode, and the body object from the response object for $response.body.

We use the TypeScript compiler API to generate an abstract syntax tree (AST) that represents the expression. This AST is then printed to a string and saved as a source file that Deno can execute.

Supported runtime expressions

In our demo, we support a limited set of runtime expressions:

• $statusCode: The HTTP status code of the response
• $steps.stepId.outputs.field: The output of a previous step
• $response.body#/path/to/field: A field in the response body selected by a JSON pointer

Arazzo supports many more runtime expressions, for example:

Parsing regular expressions

Regular expressions in Arazzo are used to validate string values against patterns. They’re particularly useful for validating IDs, dates, and other structured strings.

In our implementation, we handle regex patterns in the parseRegexCondition function:

This function takes three parameters:

• condition: The regex pattern to match against
• usedAssertions: A set to track which assertion functions we’ve used
• context: The runtime expression that selects the value to validate

The function generates code that:

1. Evaluates the context expression to get the value to validate
2. Creates a new RegExp object from the pattern
3. Uses Deno’s assertMatch function to validate the value against the pattern

The generated code looks like this:

This approach has several advantages:

• It preserves the original pattern’s flags (like i for case-insensitive matching).
• It provides clear error messages when validation fails.
• It integrates well with Deno’s testing framework.

In a production implementation, we’d want to add:

• Validation of the regex pattern syntax
• Support for named capture groups
• Error handling for malformed patterns
• Performance optimizations like pattern caching

But for our demo, this simple implementation is sufficient to show how regex validation works in Arazzo.

Parsing JSONPath expressions

JSONPath expressions are a powerful way to query JSON data. In Arazzo, we use them in success criteria to select objects or values from complex response structures. While JSON Pointer (which we’ll cover next) is great for accessing specific values, JSONPath shines when you need to:

• Validate arrays (for example, checking array length)
• Filter elements (for example, finding items matching a condition)
• Access deeply nested data with wildcards
• Aggregate values (for example, counting matches)

Here’s how our test generator handles JSONPath expressions:

This function generates code that evaluates a JSONPath expression against a context object (usually the response body). For example, given this success criterion:

Our generator creates a test that:

1. Extracts the links array from the response body using a JSON pointer
2. Evaluates the JSONPath expression $.length against this array
3. Compares the result to the expected value 5

The generated test code looks something like this:

JSONPath is particularly useful for validating:

• Array operations: $.length, $[0], $[(@.length-1)]
• Deep traversal: $..name (all name properties at any depth)
• Filtering: $[?(@.status=="active")] (elements where status is active)
• Wildcards: $.*.name (name property of all immediate children)

A few things to keep in mind when using JSONPath:

1. JSONPath isn’t well standardized, so different implementations vary widely. Arazzo makes provisions for this by allowing us to specify the JSONPath version in the test specification.
2. Even though we can specify a version, we still need to be cautious when using advanced features. Some features might not be supported by the chosen JSONPath library.
3. Check the JSONPath comparison  page to see how different libraries handle various features, and decide which features are safe to use.

Parsing JSON Pointers

While JSONPath is great for complex queries, JSON Pointer (RFC 6901)  is perfect for directly accessing specific values in a JSON document. In Arazzo, we use JSON Pointers in runtime expressions to extract values from responses and pass them to subsequent steps.

Here’s how our test generator handles JSON Pointers:

This function parses runtime expressions that use JSON Pointers. For example, given this output definition:

Our generator creates code that:

1. Takes the part after # as the JSON Pointer (/robotId)
2. Converts the pointer segments into property access expressions
3. Generates code to extract the value

The generated test code looks something like this:

Generating end-to-end tests

Now that we understand how to parse Arazzo documents, let’s look at how we generate executable tests from them. Our generator creates type-safe test code using TypeScript’s factory methods rather than string templates, providing better error detection and maintainability.

Test structure

The generator creates a test suite for each workflow in the Arazzo document. Each step in the workflow becomes a test case that executes sequentially.

Let’s explore the structure of a generated test case.

We start by setting up a test suite for the workflow, using the Arazzo workflow description as the suite name.

Next we define the serverUrl, apiKey, and context variables. The serverUrl points to the API server. We use the servers list in the OpenAPI document to determine the server URL.

We also set up the apiKey for authentication. In our demo, we use a hardcoded API key, but in a real-world scenario, we’d likely get this after authenticating with the API.

We’ll use the context object to store values extracted from the response body for use in subsequent steps.

For each step in the workflow, we generate a test case that executes the step and validates the success criteria.

Our first step is to create a new robot design session.

The HTTP method and path are extracted from the OpenAPI document using the operationId from the Arazzo step.

We set up the request headers, including the x-api-key header for authentication.

The request body is set up using the requestBody object from the Arazzo step.

We extract the response body as JSON.

We assert the success criteria for the step.

Finally, we extract the outputs from the step and store them in the context object for use in subsequent steps.

This structure repeats for each step in the workflow, creating a series of test cases that execute the workflow sequentially. The generated tests validate the API’s behavior at each step, ensuring that the workflow progresses correctly.

Future development and improvements

Our generated tests are a good start, but they might not be truly end-to-end if we don’t consider the interfaces our users interact with to access the API.

Testing with SDKs

In our demo, we use the fetch API to interact with the Build-a-bot API. While this is a common approach, it’s not always the most user-friendly. Developers often prefer SDKs that provide a more idiomatic interface to the API.

To make our tests more end-to-end, we could use the SDK Speakeasy created from the OpenAPI document to interact with the API.

Since the SDK is generated from the OpenAPI document, with names and methods derived from the API’s tags and operation IDs, we could use Arazzo to validate the SDK’s behavior against the API’s capabilities.

For example, we could:

1. Get the operationId from the Arazzo step and derive the corresponding SDK method import.
2. Call the SDK method with the required parameters.
3. Validate the response against the success criteria.
4. Extract the outputs from the response and store them in the context object.
5. Repeat for each step in the workflow.

This approach would provide a more realistic end-to-end test, validating the SDK’s behavior against the API’s capabilities.

Handling authentication

In our demo, we use a hard-coded API key for authentication. In a real-world scenario, we’d likely need to authenticate with the API to get a valid API key.

OpenAPI also supports more advanced authentication schemes like OAuth 2.0, JWT, and API key in headers, query parameters, or cookies. Our test generator should handle these schemes to ensure the tests are realistic and cover all authentication scenarios.

Arazzo can point to the security schemes in the OpenAPI document, allowing us to extract the required authentication parameters and set them up in the test suite.

Hardening the parsers against vulnerabilities

Our parsers are simple and work well for the demo, but they lack robust error handling and edge case coverage.

For example, JSONPath-plus, the library we use for JSONPath, recently fixed a remote code execution vulnerability. We should ensure our parser is up to date and secure against similar vulnerabilities, or limit the JSONPath features we support to reduce the attack surface.

This applies to parsers in general, and the risk is even higher when parsing user input and generating code from it.

Deno provides some protection by limiting access to the filesystem and network by default, but the nature of API testing means we need to access the network and read files.

Where to next?

The Arazzo specification, although released as v1.0.0, is in active development. The OpenAPI team is working on a JSON Schema for Arazzo, which will provide a formal definition of the specification’s structure and constraints.

We found the specification slightly ambiguous in places, but the team is active on GitHub  and open to feedback and contributions. If you’re interested in API testing, Arazzo is a great project to get involved with.

At Speakeasy, we’re building tools to make API testing easier and more effective. Our TypeScript, Python, and Go SDK generators can already generate tests from OpenAPI documents, and we’re working on integrating Arazzo support. Our CLI can already lint Arazzo documents, and we’ll have more to share soon.

We’re excited to see how Arazzo evolves and how it can help developers build robust, end-to-end tests for their APIs.