Versioning and evolution

Once an API has been designed, built, deployed, and integrated with by various consumers, changing the API can be very difficult.

The scale of difficulty depends on the type of changes:

• Additive changes: Adding new endpoints, adding new properties to a response, and introducing optional parameters are non-breaking changes that can typically be implemented without significant issues.
• Breaking changes: Removing or renaming endpoints, removing required fields, and changing response structures are considered breaking changes. These have the potential to disrupt existing client applications, resulting in errors and loss of functionality. Clients, especially paying customers, may face the expensive and time-consuming task of adapting their code to accommodate breaking changes.

For effective management of changes, developers must navigate the versioning and evolution of APIs carefully, ensuring that client integrations aren’t negatively impacted. Let’s explore these challenges in more detail.

When API changes are an issue

Some APIs are built purely to service a single application. For example, an API might be the backend for a web application handled by a single full-stack developer or by a team that manages both the frontend and the API. In this case, changes wouldn’t be problematic because both the frontend and backend could be updated simultaneously, ensuring that there’s no risk of breaking integration.

However, in most cases APIs are consumed by multiple clients, ranging from other teams within the same organization to external customers. This introduces complexities:

• Internal clients: Even when the consumers are within the same organization, changes may require coordination, especially if the API is used by different teams working on separate services. The timing of updates and changes can cause delays or disruptions.
• External clients: If the API is being used by third-party clients, particularly paying customers, changes can become even more difficult. External clients may resist updates due to the effort and risk involved in modifying their integrations. A major change could result in lost business, dissatisfaction, or churn.

When API consumers aren’t in sync with the development team, managing versioning becomes essential to ensuring smooth transitions.

Why APIs need to change

It’s unlikely that anyone has ever released software and thought, “That’s perfect, no change needed.”

APIs evolve over time like any other software, whether it’s due to changing business requirements, feedback from users, or the need to adopt new technologies. APIs are rarely “perfect” and immutable.

Just like software, APIs require a versioning system to accommodate changes. Developers use version numbers to signify changes in the API contract, allowing consumers to choose which version of the API they wish to use. This ensures backward compatibility for existing clients while introducing improvements and fixes in newer versions.

With most software, users can have any version running, resulting in multiple versions of the software running on various users’ computers at once. Common conventions, including Semantic Versioning , use three numbers: major, minor, and patch. So, some users might be running 1.0.0 while others run 1.0.2, and eventually, some may run 2.1.3.

A breaking change might look like:

• A change to the endpoint structure
• Adding a new required field
• Removing a field from a response
• Changing the behavior of an endpoint
• Changing validation rules
• Modifying the response format (for example, implementing a standard data format like JSON:API)

If any of this is done, a new API version may be required to avoid breaking existing clients.

Versioning an API

API versioning involves assigning a version number or identifier to the API, essentially creating multiple different APIs that are segmented in some clear way. Versioning allows consumers to specify which version of the API they wish to interact with.

There are countless ways people have tried to solve this problem over time, but there are two main approaches:

URL versioning

One of the most common approaches, URL versioning, segments the API by including a version number in the URL. Typically, only a major version is used, as seen in this example:

In this example, v1 refers to the version of the API. This initial version preserves the way the resource was designed at first.

As the API grows, a new version is introduced to accommodate changes, separate from the first version.

Here, the v2 endpoint introduces a few notable changes:

• The developers phased out auto-incrementing IDs, as per the security advice.
• They ditched the fallacy of people having two names .
• They helped users enter a preferred name to show publicly, instead of forcing everyone to publicize their legal name.

Great, but this is a big change.

If these changes were deployed on the /v1 version, they would have broken most clients’ usage: Users would see 404 errors using their old IDs, and the field changes would cause validation failures.

As the changed API runs simultaneously under /v2, both versions can be used at once. This allows clients to migrate at their own pace and developers to update the API without breaking existing clients.

Media-type versioning

Instead of embedding the version number in the URL, media-type versioning places the versioning information in the HTTP Accept header. This allows for more flexible management of the API contract without altering the URL structure.

In this case, the client specifies the version they want by including the Accept header with the version identifier (for example, application/vnd.acme.v2+json). The advantage is that the API URL remains clean, and the versioning is managed through the HTTP headers.

This approach is less common than URL versioning and has a few downsides. It’s a bit more complex to implement, and it’s not as easy for clients to see which version of the API they’re using.

API evolution as an alternative

While versioning is a popular solution to managing API changes, API evolution is an alternative that focuses on maintaining backward compatibility and minimizing breaking changes. Instead of introducing entirely new versions, API developers evolve the existing API to accommodate new requirements, but do so in a way that doesn’t disrupt clients.

API evolution is the concept of striving to maintain the “I” in API - interface elements like the request/response body, query parameters, and general functionality - only breaking them when absolutely necessary. It’s the idea that API developers bend over backwards to maintain a contract, no matter how annoying that might be. It’s often more financially and logistically viable for API developers to bear this load than to dump the workload on a wide array of consumers.

API evolution in practice

To work on a realistic example, consider this simple change:

The property name exists, and needs to be split into first_name and last_name to support Stripe’s name requirements.

It’s a minor example, but removing name and requiring all consumers to change their code to use the two new fields would be a breaking change. Let’s see how we can retain backward compatibility.

Most web application frameworks commonly used to build APIs have a feature like “serializers”, where database models are turned into JSON objects to be returned with all sensitive fields removed and any relevant tweaks or structure added.

The database might have changed to using first_name and last_name, but the API does not need to remove the name property. Instead, name can be replaced with a “dynamic property” that joins the first and last names together and is then returned in the JSON.

When a POST or PATCH is sent to the API, the API doesn’t need to think about a version number to notice whether name has been sent. If name was sent, it can split the property, and if first_name and last_name were sent, it can handle the properties as expected.

A lot of changes can be handled by introducing new properties and supporting old properties indefinitely, but at a certain point, maintaining becomes cumbersome enough to require a bigger change.

When an endpoint is starting to feel clunky and overloaded or fundamental relationships change, developers can avoid an API rewrite by evolving the API with new resources, collections, and relationships.

Changing the domain model

Let’s consider the case of Protect Earth , a reforestation and rewilding charity with a Tree Tracker API that was in need of some fundamental changes. It used to focus on tracking the trees that were planted by recording a photo, coordinates, and other metadata to allow for sponsoring and funding tree planting.

The API originally had a /trees resource as well as an /orders resource that had a plantedTrees property. However, the charity expanded beyond planting trees to sowing wildflower meadows, rewetting peat bogs, and clearing invasive species. Instead of adding /peat and /meadows resources, the API became more generic with a /units collection.

Removing /trees or plantedTrees would break customers and stem the flow of funding.

API evolution focuses on adding new functionality without breaking existing clients, so instead of removing the /trees endpoint, the API now supports both /units and /trees, with the /trees resource simply filtering the /units based on the type field:

This change allows existing developers to continue using the /trees endpoint while new developers can use the /units endpoint. The API evolves to support new functionality without breaking existing clients.

What about the /orders that contain plantedTrees? Removing this property would be a breaking change, so Protect Earth needs a backward-compatible solution. With API evolution, there are countless options.

It’s possible to add both an old property and a new property, allowing clients to migrate at their own pace. For example, we could add a new allocatedUnits property to the /orders resource, while keeping the old plantedTrees property:

However, for orders with 20,000 trees, this change means there would be 40,000 items across two almost identical sub-arrays. This is a bit of a waste, but really, it highlights an existing design flaw. Why are these sub-arrays not paginated? And why are units embedded inside orders?

Units and orders are different resources, and it’s far easier to treat them as such. API evolution gives us a chance to fix this.

There is already a /units endpoint, so let’s use that.

This way, the /order resource is just about the order, and the /units resource is about the units. This is a more RESTful design, and it’s a better way to handle the relationship between orders and units.

Where did the plantedTrees property go? It was moved behind a switch and will only show up on orders for trees. All other unit types can be found on the /units link, which benefits from full pagination.

Deprecating endpoints

All this flexibility comes with a tradeoff: It’s more work to maintain two endpoints, because there may be performance tweaks and bug reports that need to be applied to both. It’s also more work to document and test both endpoints, so it’s a good idea to keep an eye on which endpoints are being used and which aren’t, and to remove the old ones when they’re no longer needed.

Old endpoints can be deprecated using the Sunset header.

Adding a Sunset header to /trees communicates to API consumers that the endpoint will be removed. If this is done with sufficient warning and with a clear migration path, it can lead to a smooth transition for clients.

Further details can be provided in the form of a URL in a Link header and the rel="sunset" attribute.

The Link URL could direct users to a blog post or an upgrade guide in your documentation.

Deprecating properties

Deprecating properties is more challenging and generally best avoided whenever possible. You can’t use Sunset to communicate that a property is being deprecated, because it only applies to endpoints. However, OpenAPI can help.

The OpenAPI Specification version 3.1 added the deprecate keyword to allow API descriptions to communicate deprecations as an API evolves.

This change shows up in the documentation and can be used by SDKs to warn developers that they’re using a deprecated property.

Protect Earth could remove the plantedTrees property from the API entirely, but that would cause a breaking change, and it’s best to avoid breaking changes whenever possible.

A better option is to stop putting the plantedTrees property in new orders, starting on the deprecated date, and to leave it on older orders.

Protect Earth is also adding the concept of orders expiring to its API. To prevent wasting unnecessary emissions, companies need to get their data out of the API within six months, or their information will be archived. If plantedTrees isn’t added to new orders, and old orders are archived after six months, plantedTrees will eventually disappear completely and can then be removed from code.

API design-first reduces change later

Some APIs have kept their v1 APIs going for over a decade, which suggests they probably didn’t need API versioning in the first place.

Some APIs are on v14 because the API developers didn’t reach out to any stakeholders to ask what they needed out of their API. They just wrote loads of code, rushing to rewrite it every time a new consumer came along with slightly different needs, instead of finding a solution that worked for everyone.

Doing more planning, research, upfront API designing, and prototyping can cut out the need for the first few versions, as many initial changes are the result of not getting enough user and market research done early on. This is common in startups that are moving fast and breaking things, but it can happen in businesses of all sizes.

Summary

When it comes to deciding between versioning and evolution, consider how many consumers will need to upgrade, and how long that work is likely to take. If it’s two days of work, and there are 10 customers, then that’s 160 person-hours. With 1,000 customers, that’s 16,000 person-hours.

At a certain point, it becomes unconscionable to ask paying customers to do that much work, and it’s better to see whether the update could be handled with a new resource, new properties, or other backward-compatible changes that slowly phase out their older forms over time, even if it’s more work.