End-to-end API testing with Arazzo, TypeScript, and Deno
Brian Flad
October 30, 2024
We've previously written about the importance of building contract & integration tests to comprehensively cover your API's endpoints, but there's still a missing piece to the puzzle. Real users don't consume your API one endpoint at a time - they implement complex workflows that chain multiple API calls together.
That's why reliable end-to-end API tests are an important component of the testing puzzle. For your APIs most common workflows, you need to ensure that the entire process works as expected, not just the individual parts.
In this tutorial, we'll build a test generator that turns Arazzo specifications into executable end-to-end tests. You'll learn how to:
- Generate tests that mirror real user interactions with your API
- Keep tests maintainable, even as your API evolves
- Validate complex workflows across multiple API calls
- Catch integration issues before they reach production
We'll use a simple "Build-a-bot" API as our example, but the principles and code you'll learn apply to any REST API.
Arazzo? What & Why
Arazzo is a specification that describes how API calls should be sequenced to achieve specific outcomes. Think of it as OpenAPI for workflows - while OpenAPI describes what your API can do, Arazzo describes how to use it effectively.
Arazzo was designed to bridge the gap between API reference documentation and real-world usage patterns. Fortunately for us, it also makes a perfect fit for generating end-to-end test suites that validate complete user workflows rather than isolated endpoints.
By combining these specifications, we can generate tests that validate not just the correctness of individual endpoints, but the entire user journey.
Arazzo?
Arazzo roughly translates to "tapestry" in Italian. Get it? A tapestry of API calls "woven" together to create a complete user experience. We're still undecided about how to pronounce it, though. The leading candidates are "ah-RAT-so" (like fatso) and "ah-RAHT-zoh" (almost like pizza, but with a rat). There is a minor faction pushing for "ah-razzo" as in razzle-dazzle. We'll let you decide.
Let's look at a simplified (and mostly invalid) illustrative example. Imagine a typical e-commerce API workflow:
Arazzo allows us to define these workflows, and specify how each step should handle success and failure conditions, as well as how to pass data between steps and even between workflows.
From specification to implementation
The example above illustrates the concept, but let's dive into a working implementation. We'll use a simplified but functional example that you can download and run yourself. Our demo implements a subset of the Arazzo specification, focusing on the most immediately valuable features for E2E testing.
We'll use the example of an API called Build-a-bot, which allows users to create and manage their own robots. You can substitute this with your own OpenAPI document, or use the Build-a-bot API to follow along.
Arazzo deep dive
Let's examine the key components of an Arazzo specification using our Build-a-bot API example. The specification describes a workflow for creating, assembling, and activating a robot - a perfect example of a complex, multi-step process that would be difficult to test with traditional approaches.
The header identifies this as an Arazzo 1.0.0 specification and references the OpenAPI document. This connection between Arazzo and OpenAPI is crucial - it allows us to validate that the workflow steps match the API's capabilities.
Each workflow has a unique ID and can specify required inputs. In this case, we need an API key for authentication.
Let's look at the first step in our workflow. This step demonstrates several key Arazzo features.
Each step has a unique stepId
, and a description
. The stepId
is particularly important, as it allows us to reference this step's outputs in subsequent steps.
Each step also specifies an operationId
, which corresponds to an operation in the OpenAPI document. This connection ensures that the workflow steps are always in sync with the API's capabilities.
A step can define a list of parameters
that should be passed to the operation. These parameters can be static values, references to outputs from previous steps, runtime expressions that reference many other variables, or reusable objects.
In this example, the x-api-key
header is required for authentication. The value for the header is gathered at runtime from the workflow's inputs, using a runtime expression: $inputs.BUILD_A_BOT_API_KEY
. We'll explore other available expressions later.
Next up, the step defines a requestBody
object. This object specifies the request body for the operation, which is required for creating a new robot design session.
Because the request body doesn't specify a contentType
, Arazzo will look up the content-type
in the OpenAPI document for the operation.
This brings us to the meat of the step, the successCriteria
list. This list defines the conditions that must be met for the step to be considered successful.
Each success criterion in the step serves a specific purpose.
First, we validate the HTTP status code. This is a basic but essential check - the robot-creation endpoint should return 201 Created
.
Next, we validate the robotId
using a regex pattern. This ensures the ID follows the expected UUID v4 format. Notice how we use the context
field to specify which part of the response to validate, and the type
field to indicate we're using a regex pattern.
The next three criteria validate specific fields in the response body using direct comparisons. These ensure the robot is created with the correct model, name, and initial status.
Finally, we use JSONPath to validate the structure of the links
array. The condition $.length == 5
checks that exactly five links are returned.
After successful validation, the step defines its outputs and next action.
The outputs
section extracts the robotId
from the response body using a JSON pointer. This ID will be available to subsequent steps through the runtime expression $steps.createRobot.outputs.robotId
.
The onSuccess
action specifies that after successful validation, the workflow should proceed to the addParts
step. This explicit flow control helps maintain clear step sequencing.
The header identifies this as an Arazzo 1.0.0 specification and references the OpenAPI document. This connection between Arazzo and OpenAPI is crucial - it allows us to validate that the workflow steps match the API's capabilities.
Each workflow has a unique ID and can specify required inputs. In this case, we need an API key for authentication.
Let's look at the first step in our workflow. This step demonstrates several key Arazzo features.
Each step has a unique stepId
, and a description
. The stepId
is particularly important, as it allows us to reference this step's outputs in subsequent steps.
Each step also specifies an operationId
, which corresponds to an operation in the OpenAPI document. This connection ensures that the workflow steps are always in sync with the API's capabilities.
A step can define a list of parameters
that should be passed to the operation. These parameters can be static values, references to outputs from previous steps, runtime expressions that reference many other variables, or reusable objects.
In this example, the x-api-key
header is required for authentication. The value for the header is gathered at runtime from the workflow's inputs, using a runtime expression: $inputs.BUILD_A_BOT_API_KEY
. We'll explore other available expressions later.
Next up, the step defines a requestBody
object. This object specifies the request body for the operation, which is required for creating a new robot design session.
Because the request body doesn't specify a contentType
, Arazzo will look up the content-type
in the OpenAPI document for the operation.
This brings us to the meat of the step, the successCriteria
list. This list defines the conditions that must be met for the step to be considered successful.
Each success criterion in the step serves a specific purpose.
First, we validate the HTTP status code. This is a basic but essential check - the robot-creation endpoint should return 201 Created
.
Next, we validate the robotId
using a regex pattern. This ensures the ID follows the expected UUID v4 format. Notice how we use the context
field to specify which part of the response to validate, and the type
field to indicate we're using a regex pattern.
The next three criteria validate specific fields in the response body using direct comparisons. These ensure the robot is created with the correct model, name, and initial status.
Finally, we use JSONPath to validate the structure of the links
array. The condition $.length == 5
checks that exactly five links are returned.
After successful validation, the step defines its outputs and next action.
The outputs
section extracts the robotId
from the response body using a JSON pointer. This ID will be available to subsequent steps through the runtime expression $steps.createRobot.outputs.robotId
.
The onSuccess
action specifies that after successful validation, the workflow should proceed to the addParts
step. This explicit flow control helps maintain clear step sequencing.
This combination of validation patterns, data extraction, and flow control creates the foundation for testing complex API workflows.
Each success criterion serves a specific purpose in ensuring the API behaves as expected, while the outputs and success actions enable smooth workflow progression.
We'll explore the concepts we covered in more detail in the next sections, but first, let's set up a development environment so we can see Arazzo in action.
Setting up the development environment
First, clone the demo repository:
git clone https://github.com/speakeasy-api/e2e-testing-arazzo.gitcd e2e-testing-arazzo
You'll need Deno v2 (opens in a new tab) installed. On macOS and Linux, you can install Deno using the following command:
curl -fsSL https://deno.land/install.sh | sh
The repository contains:
- A simple API server built with @oak/acorn (opens in a new tab) that serves as the Build-a-bot API (in
packages/server/server.ts
) - An Arazzo specification file (
arazzo.yaml
) - An OpenAPI specification file (
openapi.yaml
) - The test generator implementation (
packages/arazzo-test-gen/generator.ts
) - Generated E2E tests (
tests/generated.test.ts
) - An SDK created by Speakeasy to interact with the Build-a-bot API (
packages/sdk
)
Running the Demo
To run the demo, start the API server:
deno task server
Deno will install the server's dependencies, then start the server on http://localhost:8080
. You can test the server by visiting http://localhost:8080/v1/robots
, which should return a 401 Unauthorized
error:
{ "status": 401, "error": "Unauthorized", "message": "Header x-api-key is required"}
Next, in a new terminal window, generate the E2E tests:
deno task dev
After installing dependencies, this command will generate the E2E tests in tests/generated.test.ts
and watch for changes to the Arazzo specification file.
You can run the tests in a new terminal window:
deno task test
This command will run the generated tests against the API server:
> deno task test
Task test deno test --allow-read --allow-net --allow-env --unstable tests/
running 1 test from ./tests/generated.test.ts
Create, assemble, and activate a new robot ...
Create a new robot design session ... ok (134ms)
Add parts to the robot ... ok (2ms)
Assemble the robot ... ok (1ms)
Configure robot features ... ok (2ms)
Activate the robot ... ok (3ms)
Get the robot details ... ok (1ms)
Create, assemble, and activate a new robot ... ok (143ms)
ok | 1 passed (6 steps) | 0 failed (147ms)
Beautiful, everything works! Let's see how we got here.
Building an Arazzo test generator
Let's start with the overall structure of the test generator.
Project structure
The test generator is a Deno project that consists of several modules, each with a specific responsibility:
-
generator.ts
: The main entry point that orchestrates the test generation process. It reads the Arazzo and OpenAPI specifications, validates their compatibility, and generates test cases. -
readArazzoYaml.ts
andreadOpenApiYaml.ts
: Handle parsing and validation of the Arazzo and OpenAPI specifications respectively. They ensure the specifications are well-formed and contain all required fields. -
expressionParser.ts
: A parser for runtime expressions like$inputs.BUILD_A_BOT_API_KEY
and$steps.createRobot.outputs.robotId
. These expressions are crucial for passing data between steps and accessing workflow inputs. -
successCriteria.ts
: Processes the success criteria for each step, including status code validation, regex patterns, direct comparisons, and JSONPath expressions. -
generateTestCase.ts
: Takes the parsed workflow and generates the actual test code, including setup, execution, and validation for each step. -
security.ts
: Handles security-related aspects like API key authentication and other security schemes defined in the OpenAPI specification. -
utils.ts
: Contains utility functions for common operations like JSON pointer resolution and type checking.
The project also includes a runtime-expression
directory containing the grammar definition for runtime expressions:
runtimeExpression.peggy
: A Peggy grammar file that defines the syntax for runtime expressionsruntimeExpression.js
: The generated parser from the grammarruntimeExpression.d.ts
: TypeScript type definitions for the parser
Let's dive deeper into each of these components to understand how they work together to generate effective E2E tests.
Parsing until you parse out
While our project says "test generator" on the tin, the bulk of our work will go into parsing different formats. To generate tests from an Arazzo document, we need to parse:
- The Arazzo document
- The OpenAPI document
- Conditions in the Arazzo success criteria
- Runtime expressions in the success criteria, outputs, and parameters
- Regular expressions in the success criteria
- JSONPath expressions in the success criteria
- JSON pointers in the runtime expressions
We won't cover all of these in detail, but we'll touch on each to get a sense of the complexity involved and the tools we use to manage it.
Parsing the Arazzo specification
The first step in our test generator is parsing the Arazzo specification in readArazzoYaml.ts
. This module reads the Arazzo YAML file and should ideally validate its structure against the Arazzo specification.
For our demo, we didn't implement full validation, instead parsing the YAML file into a JavaScript object. We then use TypeScript interfaces to define the expected structure of the Arazzo document:
export interface ArazzoDocument { arazzo: string; info: ArazzoInfo; sourceDescriptions: Array<ArazzoSourceDescription>; workflows: Array<ArazzoWorkflow>; components: Record<string, unknown>;}export interface ArazzoWorkflow { workflowId: string; description: string; inputs: { type: string; properties: Record<string, { type: string; description: string }>; }; steps: Array<ArazzoStep>;}export interface ArazzoStep { stepId: string; description: string; operationId: string; parameters?: Array<ArazzoParameter>; requestBody?: ArazzoRequestBody; successCriteria: Array<ArazzoSuccessCriterion>; outputs?: Record<string, string>;}
These TypeScript interfaces help with autocompletion, type checking, and documentation, making it easier to work with the parsed Arazzo document in the rest of our code.
The real complexity comes in validating that the parsed document follows all the rules in the Arazzo specification. For example:
- Each
workflowId
must be unique within the document - Each
stepId
must be unique within its workflow - An
operationId
must reference a valid operation in the OpenAPI document - Runtime expressions must follow the correct syntax
- Success criteria must use valid JSONPath or regex patterns
We don't validate all these rules in our demo, but in production, we'd use Zod (opens in a new tab) or Valibot (opens in a new tab) to enforce these constraints at runtime and provide helpful error messages when the document is invalid.
The OpenAPI team hasn't finalized the Arazzo specification's JSON Schema yet, but once they do, we can use it to validate the Arazzo document against the schema with tools like Ajv (opens in a new tab).
Speakeasy also provides a command-line interface for linting Arazzo documents:
# Expects arazzo.yaml in the current directoryspeakeasy lint arazzo
Parsing the OpenAPI specification
The OpenAPI specification's path is gathered from the Arazzo document. In our test, we simply use the first sourceDescription
to find the OpenAPI document path. But in a production generator, we'd need to handle multiple sourceDescriptions
and ensure the OpenAPI document is accessible.
We parse the OpenAPI document in readOpenApiYaml.ts
and use TypeScript interfaces from the npm:openapi-types
(opens in a new tab) package to define the expected structure of the OpenAPI document.
We won't cover the OpenAPI parsing in detail, but it's similar to the Arazzo parsing: Read the YAML file, parse it into a JavaScript object, and type check it against TypeScript interfaces.
For OpenAPI, writing a custom validator is more complex due to the specification's size and complexity. We recommend validating against the official OpenAPI 3.1.0 JSON Schema (opens in a new tab) using Ajv (opens in a new tab), or Speakeasy's own OpenAPI linter:
speakeasy lint openapi --schema openapi.yaml
Parsing success criteria
This is where things get interesting. Success criteria in Arazzo are a list of conditions that must be met for a step to be considered successful. Each criterion can be one of the following types:
simple
: Selects a value with a runtime expression and compares it to an expected valuejsonpath
: Selects a value using a JSONPath expression and compares it to an expected valueregex
: Validates a value against a regular expression patternxpath
: Selects a value using an XPath expression and compares it to an expected value, used for XML documents
In our demo, we don't implement the xpath
type, but we do cover the other three. Here's an example of a success criterion in the Arazzo document:
successCriteria: - condition: $statusCode == 201 - condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i context: $response.body#/robotId type: regex - condition: $response.body#/model == "humanoid" - condition: $response.body#/name == "MyFirstRobot" - condition: $response.body#/status == "designing" - context: $response.body#/links condition: $.length == 5 type: jsonpath
The condition
field is required, and contains the expression to evaluate, while the context
field specifies the part of the response to evaluate. The type
field indicates the type of validation to perform.
If no type
is specified, the success criterion is treated as a simple
comparison, where the condition
is evaluated directly.
Evaluating simple criteria
Here's an example of how we parse a simple
success criterion:
condition: $statusCode == 201
We split this simple condition into:
- Left-hand side:
$statusCode
- Runtime expression to evaluate - Operator:
==
- Comparison operator or assertion - Right-hand side:
201
- Expected value
We'll evaluate the runtime expression $statusCode
and compare it to the expected value 201
. If the comparison is true, the criterion passes; otherwise, it fails.
Runtime expressions can also reference other variables, like $inputs.BUILD_A_BOT_API_KEY
or $steps.createRobot.outputs.robotId
, or fields in the response body, like $response.body#/model
.
We'll cover runtime expressions in more detail later.
Evaluating JSONPath criteria
For JSONPath criteria, we use the jsonpath
type and a JSONPath expression to select a value from the response:
context: $response.body#/linkscondition: $.length == 5type: jsonpath
Let's break down the JSONPath criterion:
context
:$response.body#/links
- Runtime expression to select thelinks
array from the response bodycondition
:$.length == 5
- JSONPath expression compared to an expected valuetype
:jsonpath
- Indicates the criterion type
We further need to break down the condition into:
- Left-hand side:
$.length
- JSONPath expression to evaluate - Operator:
==
- Comparison operator - Right-hand side:
5
- Expected value
We evaluate the JSONPath expression $.length
and compare it to the expected value 5
. If the comparison is true, the criterion passes.
Evaluating regex criteria
For regex criteria, we use the regex
type and a regular expression pattern to validate a value:
context: $response.body#/robotIdcondition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/itype: regex
Let's break down the regex criterion:
context
:$response.body#/robotId
- Runtime expression to select therobotId
field from the response bodycondition
:/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
- Regular expression pattern to validate the value as a UUID v4type
:regex
- Indicates the criterion type
We evaluate the runtime expression $response.body#/robotId
against the regular expression pattern. If the value matches the pattern, the criterion passes.
In our implementation, we use TypeScript's factory.createRegularExpressionLiteral
to create a regular expression literal from the pattern string. This ensures that the pattern is properly escaped and formatted as a valid JavaScript regular expression.
The generated test code looks something like this:
assertMatch( response.body.robotId, new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i), 'robotId should be a valid UUID v4');
This code uses Deno's built-in assertMatch
function to validate that the robotId
matches the UUID v4 pattern. If the value doesn't match, the test fails with a helpful error message.
Parsing runtime expressions
Runtime expressions are used throughout the Arazzo specification to reference variables, fields in the response body, or outputs from previous steps. They follow a specific syntax defined in the Arazzo specification as an ABNF (augmented Backus–Naur form) grammar.
To parse runtime expressions, we use a parser generated from the ABNF grammar. In our demo, this is a two-step process. First, we use the abnf (opens in a new tab) npm package to generate a Peggy grammar file from the ABNF grammar:
cd packages/arazzo-test-genabnf_gen runtime-expression/runtimeExpression.abnf
This generates a runtime-expression/runtimeExpression.peggy
file that defines the syntax for runtime expressions. We then use the peggy (opens in a new tab) npm package to generate a parser from the Peggy grammar:
cd packages/arazzo-test-genpeggy --dts --output runtime-expression/runtimeExpression.js --format es runtime-expression/runtimeExpression.peggy
This generates a runtime-expression/runtimeExpression.js
file that contains the parser for runtime expressions. We also generate TypeScript type definitions in runtime-expression/runtimeExpression.d.ts
.
The parser reads a runtime expression like $response.body#/robotId
and breaks it down into tokens. We then evaluate the tokens to resolve the expression at runtime.
Evaluating runtime expressions
Once we've parsed a runtime expression, we need to evaluate it to get the value it references. For example, given the expression $response.body#/robotId
, we need to extract the robotId
field from the response body.
The evaluateRuntimeExpression
function in utils.ts
handles this evaluation. Here's an example of how it works:
switch (root) { case "$statusCode": { // Handle $statusCode expressions result = factory.createPropertyAccessExpression( factory.createIdentifier("response"), factory.createIdentifier("status"), ); break; } case "$response.": { // Handle $request and $response expressions const data = factory.createIdentifier("data"); // use parseRef to handle everything after $response.body const pointer = parsePointer(expression.slice(15)); result = pointer.length > 0 ? factory.createPropertyAccessExpression(data, pointer.join(".")) : data; break; } // Handle other cases ...}
Here, we handle two types of runtime expressions: $statusCode
and $response.body
. We extract the status
field from the response
object for $statusCode
, and the body
object from the response
object for $response.body
.
We use the TypeScript compiler API to generate an abstract syntax tree (AST) that represents the expression. This AST is then printed to a string and saved as a source file that Deno can execute.
Supported runtime expressions
In our demo, we support a limited set of runtime expressions:
$statusCode
: The HTTP status code of the response$steps.stepId.outputs.field
: The output of a previous step$response.body#/path/to/field
: A field in the response body selected by a JSON pointer
Arazzo supports many more runtime expressions, for example:
Expression | Description |
---|---|
$url | The full URL of the request |
$method | The HTTP method of the request |
$statusCode | The HTTP status code of the response |
$request.header.{name} | The value of the specified request header |
$request.query.{name} | The value of the specified query parameter from the request URL |
$request.path.{name} | The value of the specified path parameter from the request URL |
$request.body | The entire request body |
$request.body#/path/to/property | The value of the specified JSON pointer path from the request body |
$response.header.{name} | The value of the specified response header |
$response.body | The entire response body |
$response.body#/path/to/property | The value of the specified JSON pointer path from the response body |
$inputs.{name} | The value of the specified workflow input |
$outputs.{name} | The value of the specified workflow output |
$steps.{stepId}.{outputName} | The value of the specified output from the step with ID {stepId} |
$workflows.{id}.{inputName} | The value of the specified input from the workflow with ID {id} |
$workflows.{id}.{outputName} | The value of the specified output from the workflow with ID {id} |
Parsing regular expressions
Regular expressions in Arazzo are used to validate string values against patterns. They're particularly useful for validating IDs, dates, and other structured strings.
In our implementation, we handle regex patterns in the parseRegexCondition
function:
function parseRegexCondition( condition: string, usedAssertions: Set<string>, context: string,): Expression { usedAssertions.add("assertMatch"); return factory.createCallExpression( factory.createIdentifier("assertMatch"), undefined, [ evaluateRuntimeExpression(context), factory.createNewExpression( factory.createIdentifier("RegExp"), undefined, [factory.createRegularExpressionLiteral(condition)], ), factory.createStringLiteral(condition), ], );}
This function takes three parameters:
condition
: The regex pattern to match againstusedAssertions
: A set to track which assertion functions we've usedcontext
: The runtime expression that selects the value to validate
The function generates code that:
- Evaluates the context expression to get the value to validate
- Creates a new RegExp object from the pattern
- Uses Deno's
assertMatch
function to validate the value against the pattern
The generated code looks like this:
assertMatch( response.body.robotId, new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i), 'robotId should be a valid UUID v4');
This approach has several advantages:
- It preserves the original pattern's flags (like
i
for case-insensitive matching). - It provides clear error messages when validation fails.
- It integrates well with Deno's testing framework.
In a production implementation, we'd want to add:
- Validation of the regex pattern syntax
- Support for named capture groups
- Error handling for malformed patterns
- Performance optimizations like pattern caching
But for our demo, this simple implementation is sufficient to show how regex validation works in Arazzo.
Parsing JSONPath expressions
JSONPath expressions are a powerful way to query JSON data. In Arazzo, we use them in success criteria to select objects or values from complex response structures. While JSON Pointer (which we'll cover next) is great for accessing specific values, JSONPath shines when you need to:
- Validate arrays (for example, checking array length)
- Filter elements (for example, finding items matching a condition)
- Access deeply nested data with wildcards
- Aggregate values (for example, counting matches)
Here's how our test generator handles JSONPath expressions:
function parseJsonPathExpression(path: string, context: string): Expression { return factory.createCallExpression( factory.createIdentifier("JSONPath"), undefined, [ factory.createObjectLiteralExpression( [ factory.createPropertyAssignment( factory.createIdentifier("wrap"), factory.createFalse(), ), factory.createPropertyAssignment( factory.createIdentifier("path"), factory.createStringLiteral(path), ), factory.createPropertyAssignment( factory.createIdentifier("json"), evaluateRuntimeExpression(context), ), ], true, ), ], );}
This function generates code that evaluates a JSONPath expression against a context object (usually the response body). For example, given this success criterion:
successCriteria: - context: $response.body#/links condition: $.length == 5 type: jsonpath
Our generator creates a test that:
- Extracts the
links
array from the response body using a JSON pointer - Evaluates the JSONPath expression
$.length
against this array - Compares the result to the expected value
5
The generated test code looks something like this:
assertEquals( JSONPath({ wrap: false, path: "$.length", json: response.body.links }), 5, "links array should contain exactly 5 elements");
JSONPath is particularly useful for validating:
- Array operations:
$.length
,$[0]
,$[(@.length-1)]
- Deep traversal:
$..name
(all name properties at any depth) - Filtering:
$[?(@.status=="active")]
(elements where status isactive
) - Wildcards:
$.*.name
(name property of all immediate children)
A few things to keep in mind when using JSONPath:
- JSONPath isn't well standardized, so different implementations vary widely. Arazzo makes provisions for this by allowing us to specify the JSONPath version in the test specification.
- Even though we can specify a version, we still need to be cautious when using advanced features. Some features might not be supported by the chosen JSONPath library.
- Check the JSONPath comparison (opens in a new tab) page to see how different libraries handle various features, and decide which features are safe to use.
Parsing JSON Pointers
While JSONPath is great for complex queries, JSON Pointer (RFC 6901) (opens in a new tab) is perfect for directly accessing specific values in a JSON document. In Arazzo, we use JSON Pointers in runtime expressions to extract values from responses and pass them to subsequent steps.
Here's how our test generator handles JSON Pointers:
function evaluateRuntimeExpression(expression: string): Expression { // ... case "$response.": { const data = factory.createIdentifier("data"); // Parse everything after $response.body const pointer = parsePointer(expression.slice(15)); result = pointer.length > 0 ? factory.createPropertyAccessExpression(data, pointer.join(".")) : data; break; } // ...}
This function parses runtime expressions that use JSON Pointers. For example, given this output definition:
outputs: robotId: $response.body#/robotId
Our generator creates code that:
- Takes the part after
#
as the JSON Pointer (/robotId
) - Converts the pointer segments into property access expressions
- Generates code to extract the value
The generated test code looks something like this:
// During test setupconst context = {};// ...// In the first testconst data = response.json();// Generated because of outputs: { robotId: $response.body#/robotId } in the Arazzo document// highlight-next-linecontext["createRobot.outputs.robotId"] = data.robotId;// ...// In a subsequent testconst robotId = context["createRobot.outputs.robotId"];const response = await fetch(`${serverUrl}/v1/robots/${robotId}/assemble`, { // ...});
Generating end-to-end tests
Now that we understand how to parse Arazzo documents, let's look at how we generate executable tests from them. Our generator creates type-safe test code using TypeScript's factory methods rather than string templates, providing better error detection and maintainability.
Test structure
The generator creates a test suite for each workflow in the Arazzo document. Each step in the workflow becomes a test case that executes sequentially.
Let's explore the structure of a generated test case.
We start by setting up a test suite for the workflow, using the Arazzo workflow description
as the suite name.
Next we define the serverUrl
, apiKey
, and context
variables. The serverUrl
points to the API server. We use the servers
list in the OpenAPI document to determine the server URL.
We also set up the apiKey
for authentication. In our demo, we use a hardcoded API key, but in a real-world scenario, we'd likely get this after authenticating with the API.
We'll use the context
object to store values extracted from the response body for use in subsequent steps.
For each step in the workflow, we generate a test case that executes the step and validates the success criteria.
Our first step is to create a new robot design session.
The HTTP method and path are extracted from the OpenAPI document using the operationId
from the Arazzo step.
We set up the request headers, including the x-api-key
header for authentication.
The request body is set up using the requestBody
object from the Arazzo step.
We extract the response body as JSON.
We assert the success criteria for the step.
Finally, we extract the outputs from the step and store them in the context
object for use in subsequent steps.
We start by setting up a test suite for the workflow, using the Arazzo workflow description
as the suite name.
Next we define the serverUrl
, apiKey
, and context
variables. The serverUrl
points to the API server. We use the servers
list in the OpenAPI document to determine the server URL.
We also set up the apiKey
for authentication. In our demo, we use a hardcoded API key, but in a real-world scenario, we'd likely get this after authenticating with the API.
We'll use the context
object to store values extracted from the response body for use in subsequent steps.
For each step in the workflow, we generate a test case that executes the step and validates the success criteria.
Our first step is to create a new robot design session.
The HTTP method and path are extracted from the OpenAPI document using the operationId
from the Arazzo step.
We set up the request headers, including the x-api-key
header for authentication.
The request body is set up using the requestBody
object from the Arazzo step.
We extract the response body as JSON.
We assert the success criteria for the step.
Finally, we extract the outputs from the step and store them in the context
object for use in subsequent steps.
This structure repeats for each step in the workflow, creating a series of test cases that execute the workflow sequentially. The generated tests validate the API's behavior at each step, ensuring that the workflow progresses correctly.
Future development and improvements
Our generated tests are a good start, but they might not be truly end-to-end if we don't consider the interfaces our users interact with to access the API.
Testing with SDKs
In our demo, we use the fetch
API to interact with the Build-a-bot API. While this is a common approach, it's not always the most user-friendly. Developers often prefer SDKs that provide a more idiomatic interface to the API.
To make our tests more end-to-end, we could use the SDK Speakeasy created from the OpenAPI document to interact with the API.
Since the SDK is generated from the OpenAPI document, with names and methods derived from the API's tags and operation IDs, we could use Arazzo to validate the SDK's behavior against the API's capabilities.
For example, we could:
- Get the
operationId
from the Arazzo step and derive the corresponding SDK method import. - Call the SDK method with the required parameters.
- Validate the response against the success criteria.
- Extract the outputs from the response and store them in the
context
object. - Repeat for each step in the workflow.
This approach would provide a more realistic end-to-end test, validating the SDK's behavior against the API's capabilities.
Handling authentication
In our demo, we use a hard-coded API key for authentication. In a real-world scenario, we'd likely need to authenticate with the API to get a valid API key.
OpenAPI also supports more advanced authentication schemes like OAuth 2.0, JWT, and API key in headers, query parameters, or cookies. Our test generator should handle these schemes to ensure the tests are realistic and cover all authentication scenarios.
Arazzo can point to the security schemes in the OpenAPI document, allowing us to extract the required authentication parameters and set them up in the test suite.
Hardening the parsers against vulnerabilities
Our parsers are simple and work well for the demo, but they lack robust error handling and edge case coverage.
For example, JSONPath-plus, the library we use for JSONPath, recently fixed a remote code execution vulnerability. We should ensure our parser is up to date and secure against similar vulnerabilities, or limit the JSONPath features we support to reduce the attack surface.
This applies to parsers in general, and the risk is even higher when parsing user input and generating code from it.
Deno provides some protection by limiting access to the filesystem and network by default, but the nature of API testing means we need to access the network and read files.
Where to next?
The Arazzo specification, although released as v1.0.0, is in active development. The OpenAPI team is working on a JSON Schema for Arazzo, which will provide a formal definition of the specification's structure and constraints.
We found the specification slightly ambiguous in places, but the team is active on GitHub (opens in a new tab) and open to feedback and contributions. If you're interested in API testing, Arazzo is a great project to get involved with.
At Speakeasy, we're building tools to make API testing easier and more effective. Our TypeScript, Python, and Go SDK generators can already generate tests from OpenAPI documents, and we're working on integrating Arazzo support. Our CLI can already lint Arazzo documents, and we'll have more to share soon.
We're excited to see how Arazzo evolves and how it can help developers build robust, end-to-end tests for their APIs.