API Advice
How To Build A Best In Class Python SDK
Tristan Cartledge
April 29, 2024
This tutorial demonstrates one way to code and package a Python SDK.
Why Are We Building an SDK?
Since the 2000s, HTTP APIs have enabled businesses to be more connected than ever and, as a result, produce higher utility and profit â no wonder APIs have exploded in popularity.
So, your business has created an API? Great! At first, developers might use curl
or create a Postman collection to experiment with your API. However, the true power of an API is only unleashed when developers can interact with it in an automated and programmatic way. To get some real work done, you'll want to wrap the HTTP API in your language of choice, so that it can be testable and have the most important aspects abstracted for your convenience and productivity.
What Are We Building?
We'll build a Python SDK that wraps an HTTP API.
Fictional e-commerce startup ECOM enables the selling of products through the creation of online stores and associated products on its website. The ECOM HTTP API enables developers to create stores and manage products in an automated way. We want to give Python programmers easy access to the ECOM API with a robust, secure, and user-friendly SDK that handles the underlying HTTP API service in a type-safe, programmatic way.
Example SDK Repository
You can follow along without downloading our complete example repository, but if you get stuck or want to see the final result, you can find the code on GitHub at speakeasy-api/ecom-sdk (opens in a new tab).
Initialize the Project
Before we begin coding the SDK, let's set up our project.
Install pyenv
While not strictly necessary in all cases, we prefer to manage Python versions with pyenv. If you're building the SDK with us, install pyenv
(opens in a new tab) for your operating system.
On macOS, you can install pyenv
with Homebrew:
brew install pyenv
Decide on a Python Version
Deciding on a Python version (opens in a new tab) that your SDK will support is important. At the time of writing, Python 3.8 is the minimum version supported by many popular libraries and frameworks. Python 3.12 is the latest stable version. Even though Python 2 is no longer supported, some older projects still use it, so you may need to support Python 2.7 if your SDK is to be used in internal legacy systems that have not yet been updated.
For this tutorial, we'll use Python 3.8, because it is the oldest version that still receives security updates.
Install Python 3.8
Install Python 3.8 with pyenv
:
pyenv install 3.8.19
Set Up a Python Virtual Environment
We'll set up a Python virtual environment to isolate our project dependencies from the system Python installation.
Create a new Python 3.8 virtual environment:
pyenv virtualenv 3.8.19 ecom-sdk
Activate the virtual environment:
pyenv activate ecom-sdk
After activating the virtual environment, depending on the shell you're using, your shell prompt might change to indicate that you are now using the virtual environment. Any Python packages you install will be installed in the virtual environment and won't affect the system Python installation.
Upgrade pip
When you create a new virtual environment, it's good practice to upgrade pip
to the latest version:
python3.8 -m pip install --upgrade pip
Decide on a Build Backend
The complexities of Python packaging (opens in a new tab) can make navigating your options challenging. We will outline the options briefly before selecting an option for this particular example.
In the past, Python used a setup.py
or setup.cfg
configuration file to prepare Python packages for distribution, but new standards (like PEPs 517, 518, 621, and 660) and build tools are modernizing the Python packaging landscape. We want to configure a modern build backend for our SDK and we have many options to choose from:
- Hatchling (opens in a new tab)
- Poetry (opens in a new tab)
- PDM (opens in a new tab)
- Setuptools (opens in a new tab)
- Flit (opens in a new tab)
- Maturin (opens in a new tab)
Some factors to consider when selecting a build backend include:
- Python version support: Some build backends only support a subset of Python versions, so ensure your chosen build backend supports the correct versions.
- Features: Determine the features you need and choose a backend that meets your requirements. For example, do you need features like project management tools, or package uploading and installation capabilities?
- Extension support: Do you need support for extension modules in other languages?
We'll use Hatchling (opens in a new tab) in this tutorial for its convenient defaults and ease of configurability.
Set Up a Project With Hatchling
First, install Hatch:
pip install hatch
Now, create a new project with Hatch:
hatch new -i
The -i
flag tells Hatch to ask for project information interactively. Enter the following information when prompted:
Project name: ECOM SDKDescription: Python SDK for the ECOM HTTP API
Hatch will create a new project directory with the name ecom-sdk
and initialize it with a basic pyproject.toml
file.
Change into the project directory:
cd ecom-sdk
See the Project Structure
Run tree
to see the project structure:
tree .
The project directory should now contain the following files:
.âââ LICENSE.txtâââ README.mdâââ pyproject.tomlâââ srcâ âââ ecom_sdkâ âââ __about__.pyâ âââ __init__.pyâââ tests âââ __init__.py4 directories, 6 files
The LICENSE.txt
file contains the MIT license, the README.md
file has a short installation hint, and the pyproject.toml
file contains the project metadata.
Update the README File
If you continue to support and expand this SDK, you'll want to keep the README.md
file up to date with the latest documentation, providing installation and usage instructions, and any other information relevant to your users. Without a good README, developers won't know where to start and won't use the SDK.
For now, let's add a short description of what the SDK does:
# ECOM SDKPython SDK for the ECOM HTTP API
Create a Basic SDK
Add a ./src/ecom_sdk/sdk.py
file containing Python code, for example:
def sdkFunction(): return 1
We'll use the ./src/ecom_sdk/sdk.py
file to ensure our testing setup works.
Create a Test
As we add functionality to the SDK, we will populate the ./tests
directory with tests.
For now, create a ./src/ecom_sdk/test_sdk.py
file containing test code:
import src.ecom_sdk.sdkdef test_sdk(): assert src.ecom_sdk.sdk.sdkFunction() == 1
Later in this guide, we'll run the test with scripts provided by Hatch.
Inspect the Build Backend Configuration
The pyproject.toml
file contains the project metadata and build backend configuration.
Be sure to take a peek at the pyproject.toml
guide (opens in a new tab) and specification (opens in a new tab) for details on all the possible metadata fields available for the [project]
table. For example, you may want to enhance the discoverability of your SDK on PyPI by specifying keyword metadata or additional classifiers.
Test Hatch
Here's the Hatch test script that we'll use to run tests:
hatch run test
The first time you run the test script, Hatch will install the necessary dependencies and run the tests. Subsequent runs will be faster because the dependencies are already installed.
After running the test script, you should see output similar to the following:
========================= test session starts ==========================platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0rootdir: /speakeasy/ecom-sdkconfigfile: pyproject.tomlcollected 1 itemtests/test_sdk.py . [100%]========================== 1 passed in 0.00s ===========================
Code the SDK
Now that we have the project set up, let's start coding the SDK.
Add an SDK Method To Fetch a List of Stores
In the ./src/ecom_sdk/sdk.py
file, define a class, constructor, and list_stores()
method for fetching a list of stores from our API:
import requestsHTTP_TIMEOUT_SECONDS = 10class EComSDK: def __init__(self, api_url, api_key): self.api_url = api_url self.api_key = api_key def list_stores(self): r = requests.get( self.api_url + "/store", headers={"X-API-KEY": self.api_key}, timeout=HTTP_TIMEOUT_SECONDS, ) if r.status_code == 200: return r.json() else: raise Exception("Invalid response status code: " + str(r.status_code))
Now we can begin writing tests in the ./tests/test_sdk.py
file to test that the newly added list_stores()
method works as intended.
In requests.get()
, we use HTTP_TIMEOUT_SECONDS
to set a maximum bound for waiting for the request to finish. If we don't set a timeout, the requests
library will wait for a response forever.
Add the following to ./tests/test_sdk.py
:
from src.ecom_sdk.sdk import EComSDKimport responsesfrom responses import matchersapi_url = "https://example.com"api_key = "hunter2"def test_sdk_class(): sdk = EComSDK(api_url, api_key) assert sdk.api_url == api_url assert sdk.api_key == api_key@responses.activatedef test_sdk_list_stores(): responses.add( responses.GET, api_url + "/store", json=[ {"id": 1, "name": "Lidl", "products": 10}, {"id": 2, "name": "Walmart", "products": 15}, ], status=200, match=[matchers.header_matcher({"X-API-KEY": api_key})], ) sdk = EComSDK(api_url, api_key) stores = sdk.list_stores() assert len(stores) == 2 assert stores[0]["id"] == 1 assert stores[0]["name"] == "Lidl" assert stores[1]["id"] == 2 assert stores[1]["name"] == "Walmart"
Here we use the responses
library to mock API responses since we don't have a test or staging version of the ECOM HTTP API. However, even with the ECOM API, we might choose to have some part of the testing strategy mock the API responses as this approach can be faster and tightly tests code without external dependencies.
The test_sdk_class()
test function checks that we can instantiate the class and the correct values are set internally.
The test_sdk_list_stores()
test function makes a call to sdk.list_stores()
to test that it receives the expected response, which is a JSON array of stores associated with the user's account.
Add Dependencies
We need to add the requests
and responses
libraries to the project dependencies. Add the following to the pyproject.toml
file:
dependencies = [ "requests", "responses", "pydantic",]
Run the Tests
Let's run the Hatch test
script we configured earlier to check that everything is working correctly:
hatch run test
Once Hatch has installed our new dependencies, you should see output similar to the following:
=============================== test session starts ================================platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0rootdir: /speakeasy/ecom-sdkconfigfile: pyproject.tomlcollected 2 itemstests/test_sdk.py .. [100%]================================ 2 passed in 0.09s =================================
Exception Handling
To create a pleasant surface for our SDK users, we'll hide details of HTTP implementation behind exception handling and provide helpful error messages should things go wrong.
Let's modify the list_stores()
function to catch some more common errors and print helpful information for the user:
When an exception is raised from the call to requests.get()
, we catch the exception, add useful information, and then re-raise the exception for the SDK user to handle in their code.
The call to r.raise_for_status()
modifies the requests
library behavior to raise an HTTPError
if the HTTP response status code is unsuccessful. Read more about raise_for_status()
in the Requests library documentation (opens in a new tab).
Now we'll add tests for a 403 response and a connection error to the test_sdk.py
file. We'll also import the requests
library into the test_sdk.py
file:
This test ensures that the helpful error message from our SDK is displayed when a connection error or 403 response occurs.
Check that the new code passes the test:
hatch run test
Type Safety With Enums
To make development easier and less error-prone for our SDK users, we'll define Enums to use with the list_products
endpoint.
Add the following Enums to the EComSDK
class:
The ProductSort
and ProductSortOrder
Enums inherit from the str
class and can be used with the list_products()
method we'll define next â ProductSort
specifies the sort category, and ProductSortOrder
determines the sort order of the product list.
Now define the list_products()
function at the bottom of the EComSDK
class:
The list_products()
function accepts named parameters sort_by
and sort_order
. If the list_products()
function is called with named parameters, the parameters are added to the params
dictionary to be included in the request's HTTP query parameters.
An SDK user can call the list_products()
function with the string list_products(sort_by="quantity")
or with the safer equivalent using Enums: list_products(sort_by=sdk.ProductSort.QUANTITY)
. By encouraging the user to use Enums in this way, we prevent HTTP error requests that don't clearly identify what went wrong should a user mistype a string, for example, "prise" instead of "price".
Now add another test at the bottom of the test_sdk.py
file:
Check that the new test passes:
hatch run test
Output Type Safety With Pydantic
To make the SDK even more user-friendly, we can use Pydantic to create data models for the responses from the ECOM API.
First, add Pydantic to the project dependencies in the pyproject.toml
file:
dependencies = [ "requests", "responses", "pydantic",]
Now, create a Product
model in the ./src/ecom_sdk/models.py
file:
Next, import the Product
model into the ./src/ecom_sdk/sdk.py
file:
Modify the list_products()
function to return a list of Product
models:
The list_products()
function now returns a list of Product
models created from the JSON response.
Now update the test_sdk.py
file to test the new Product
model:
We'll also need to update the test_sdk_list_products()
test to check that the Product
models are returned from the list_products()
function.
Check that the new tests pass:
hatch run test
We've now added type safety to the SDK using Pydantic, ensuring that the SDK user receives a list of Product
models when calling the list_products()
function. The same principle can be applied to other parts of the SDK to ensure that the user receives the correct data types.
Add Type Safety to the SDK Constructor
To ensure that the SDK user provides the correct data types when instantiating the EComSDK
class, we can use Pydantic to create a Config
model for the SDK constructor.
First, create a Config
model in the ./src/ecom_sdk/models.py
file:
Next, import the Config
model into the ./src/ecom_sdk/sdk.py
file:
Now, modify the EComSDK
class to accept a Config
model in the constructor:
The EComSDK
class now accepts a Config
model in the constructor, ensuring that the SDK user provides the correct data types when instantiating the class.
Let's update the test_sdk.py
file to test the new Config
model:
Follow the same pattern to update the other tests in the test_sdk.py
file to use the new Config
model. Replace sdk = EComSDK(api_url, api_key)
with:
Check that the new tests pass:
hatch run test
Things to Consider
We've described some basic steps for creating a custom Python SDK, but there are many facets to an SDK project besides code that contribute to its success or failure. For any publicly available Python library, you should consider such aspects as documentation, linting, and licensing.
Documentation
To focus on the nuts and bolts of a custom Python SDK, this guide does not cover developing an SDK's documentation. But documentation is critical to ensure your users can easily pick up your library and use it productively. Without good documentation, developers may opt not to use your SDK at all.
All SDK functions, parameters, and behaviors should be documented, and creating beautiful, functional documentation is an essential part of making your SDK usable. You can roll your own documentation and add Markdown to the project README.md
file or use a popular library like Sphinx (opens in a new tab) that includes such features as automatic highlighting, themes, and HTML or PDF outputs.
Linting
Consistently formatted code improves readability and encourages contributions that are not jarring in the context of the existing codebase. While this guide doesn't cover linting, linting your SDK code should form part of creating an SDK.
For best results, linting should be as far-reaching and opinionated as possible, with little to zero default configuration required. Some popular options for linting include Flake8 (opens in a new tab), Ruff (opens in a new tab), and Black (opens in a new tab).
Licensing
A project's license can significantly influence the type and number of potential open-source contributors. Choosing the right license for your SDK is key. Do you want your SDK's license to be community-orientated? Simple and permissive? To preserve a particular philosophy on enforcing the sharing of code modifications? In this example, we selected the MIT license for simplicity and ease of use. If you're unsure which license would best suit your needs, consider using a tool like Choose a license (opens in a new tab).
Other Endpoints
This tutorial covered a basic SDK for two endpoints. Consider that real-world SDKs can have many more endpoints and features. You can use the same principles to add more endpoints to your SDK.
Authentication Methods
If APIs use OAuth 2.0 or other authentication methods, you'll need to add authentication methods to your SDK. You can use libraries like Authlib (opens in a new tab) to handle OAuth 2.0 authentication.
Pagination
If the API returns paginated results, you'll need to handle pagination in your SDK.
Retries and Backoff
If the API is rate-limited or has intermittent failures, you'll need to add retry and backoff logic to your SDK.
Build the Package
Now that we have a working SDK, we can build the package for distribution.
hatch build
Hatch will build the package and output the distribution files:
ââââââââââââââââââââââââââââââââââââââ sdist âââââââââââââââââââââââââââââââââââââââdist/ecom_sdk-0.0.1.tar.gzââââââââââââââââââââââââââââââââââââââ wheel âââââââââââââââââââââââââââââââââââââââdist/ecom_sdk-0.0.1-py3-none-any.whl
You should now have a ./dist
directory containing your source (.tar.gz
) and built (.whl
) distributions:
.âââ distâ âââ ecom_sdk-0.0.1-py3-none-any.whlâ âââ ecom_sdk-0.0.1.tar.gz
Upload the SDK to the Distribution Archives
We now have just enough to upload the SDK to the PyPI test repo at test.pypi.org.
To upload the build to TestPyPI, you need:
-
A test PyPI account. Go to https://test.pypi.org/account/register/ (opens in a new tab) to register.
-
A PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens (opens in a new tab), setting the "Scope" to "Entire account".
Don't close the token page until you have copied and saved the token. For security reasons, the token will only appear once.
If you plan to automate SDK publishing in your CI/CD pipeline, you should create per-project tokens. For automated releasing using CI/CD, it is recommended that you create per-project API tokens.
Finally, publish the package distribution files by executing:
hatch publish -r test
The -r test
switch specifies that we are using the TestPyPI repository.
Hatch will ask for a username and credentials. Use __token__
as the username (to indicate that we are using a token value rather than a username) and paste your PyPI API token in the credentials field.
hatch publish -r test â [15:13]âEnter your username [__token__]: __token__Enter your credentials: (paste your token here)dist/ecom_sdk-0.0.1-py3-none-any.whl ... successdist/ecom_sdk-0.0.1.tar.gz ... success[ecom-sdk]https://test.pypi.org/project/ecom-sdk/0.0.1/
Your shiny new package is now available to all your new Python customers!
The ecom-sdk
python package can now be installed using:
pip install --index-url https://test.pypi.org/simple/ --no-deps ecom-sdk
We use the --index-url
switch to specify the TestPyPI repo instead of the default live repo. We use the --no-deps
switch because the test PyPI repo doesn't have all dependencies (because it's a test repo) and the pip install
command would fail otherwise.
Automatically Create Language-Idiomatic SDKs From Your API Specification
We've taken the first few steps in creating a Python SDK but as you can see, we've barely scratched the surface. It takes a good deal of work and dedication to iterate on an SDK project and get it built right. Then comes the maintenance phase, dealing with pull requests, and triaging issues from contributors.
You might want to consider automating the creation of SDKs with a managed service like Speakeasy. You can quickly get up and running with SDKs in nine languages with best practices baked in and maintain them automatically with the latest language and security updates.