A Design Pattern for Python API Client Libraries
Recently I've been getting more and more interested in blockchain and ways it can be used in the education industry, in particular how to secure student information to prevent tampering and allow audits without exposing sensitive user data. The details of that project will come in the next post, but as part of the process I ended up writing a new Python client library for Factom, a blockchain service provider. I'm going to use this post to share some thoughts on what makes a good API client and share the patterns I follow when designing client libraries from scratch, using the Factom client as an example.
Before we get into the technical details, I first want give some background on the Factom Project. Given the obscene amount of money being blindly thrown into cryptocurrencies, it's incredible how few blockchain services with any kind of practical application exist. Factom is a refreshing exception to that trend. If you're unfamiliar with the project I encourage to check out the whitepaper, but tl;dr Factom is a service layer on top of the Bitcoin blockchain that allows arbitrary amounts of data to be secured quickly and at a fixed cost. It's a perfect fit for any application that needs an immutable records system and was the natural choice for this project.
The code for the API client is available on GitHub.
Design goals
Let's first consider the interface from a developer's perspective and then we'll take a look under the hood. These are the goals I keep in mind when designing a client library:
1. Minimal boilerplate: The whole point of a library is to reduce the number of lines a developer needs to write. Ideally we'd write zero lines of code, but as I haven't figured out how to get away with that yet, let's assume the best we can do is one line per action. One line of code gets you a client interface that's ready to go. One line of code gets you the result of a method call.
# Instantiate a new client interface and query the balance of
# Factoid tokens at our address.
>>> client = Factomd(username='rpc_username', password='rpc_password')
>>> client.factoid_balance(fct_address)
{'balance': 50000}
2. Seamless integration: As a developer, the last thing I want to have to think about is transport layer or server implementation details. Ideally, I want my API clients to look like any other local class or module interface. Methods should take native data types and spit out native data types.
# Read all of the entries in a Factom chain. Make a JSON-RPC call
# to the 'chain-head' method at localhost:8088/v2 with the chain
# ID as a parameter to determine the Key Merkle Root of the first
# data-containing entry block and I really don't want to think
# about this I just want my data...
>>> factom.read_chain(chain_id)
[{'chainid': 'da2ffed0ae7b33acc718089edc0f1d001289857cc27a49b6bc4dd22fac971495', 'extids': ['random', 'entry', 'id'], 'content': 'entry_content'}, {'chainid': 'da2ffed0ae7b33acc718089edc0f1d001289857cc27a49b6bc4dd22fac971495', 'extids': ['random', 'chain', 'id'], 'content': 'chain_content'}]
3. Sensible error handling: There's a special circle of hell reserved for libraries whose error handling mechanism consists of checking for a 4XX HTTP status code. Good error handling should make use of language control structures and error types. Python exceptions are great:
# Create a new Factom chain.
>>> walletd.new_chain(factomd, chain_id, chain_content, ec_address=ec_address)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/src/factom/client.py", line 196, in new_chain
'ecpub': ec_address or self.ec_address
File "/src/factom/client.py", line 56, in _request
handle_error_response(resp)
File "/src/factom/exceptions.py", line 18, in handle_error_response
raise codes[code](message=message, code=code, data=data, response=resp)
factom.exceptions.InvalidParams: -32602: Invalid params
# Uh oh. Let's try that again with some exception handling.
>>> try:
... walletd.new_chain(factomd, chain_id, chain_content, ec_address=ec_address)
... except InvalidParams as e:
... print(e.message)
Chain already exists
The code
Typically, I organize my API clients into 3 modules:
session.py
This is where transport layer-related code lives. For HTTP clients the session class is usually wrapped around a requests.Session object and is responsible for maintaining authentication and configuring TLS--basically anything having to do with making the actual remote request.
A very simple session.py might look like:
class FactomAPISession(requests.Session):
def __init__(self, *args, **kwargs):
super(FactomAPISession, self).__init__(*args, **kwargs)
self.headers.update({
'Accept-Charset': 'utf-8',
'Content-Type': 'text/plain',
'User-Agent': 'factom-api/{}'.format(VERSION)
})
def init_basic_auth(self, username, password):
credentials = b64encode('{}:{}'.format(username, password).encode())
self.headers.update({
'Authorization': 'Basic {}'.format(credentials.decode())
})
Here we're initializing a basic HTTP session and setting a few custom headers, as well as providing a method to configure basic HTTP authentication via username and password. Reusing the session reduces a quite a bit of boilerplate when it comes time to make actual requests.
client.py
The client module provides the interface class. It typically contains a single class that gets initialized with any configuration needed to build the API client and contains the methods for making requests.
class Factomd(object):
def __init__(self, host='localhost:8090', version='v2', username=None, password=None):
"""
Instantiate a new API client.
Args:
host (str): Hostname of the factomd instance.
version (str): API version to use. This should remain 'v2'.
username (str): RPC username for protected APIs.
password (str): RPC password for protected APIs.
"""
self.version = version
self.host = host
# Initialize the session.
self.session = FactomAPISession()
# If authentication credentials are provided, pass them
# along to the session.
if username and password:
self.session.init_basic_auth(username, password)
# Convenience method for building request URLs.
@property
def url(self):
return urljoin(self.host, self.version)
# Perform an API request.
def _request(self, method, params=None, id=0):
data = {
'jsonrpc': '2.0',
'id': id,
'method': method,
}
if params:
data['params'] = params
# Ask the session to perform a JSON-RPC request
# with the parameters provided.
resp = self.session.request('POST', self.url, json=data)
# If something goes wrong, we'll pass the response
# off to the error-handling code
if resp.status_code >= 400:
handle_error_response(resp)
# Otherwise return the result dictionary.
return resp.json()['result']
# API methods
def chain_head(self, chain_id):
return self._request('chain-head', {
'chainid': chain_id
})
def commit_chain(self, message):
return self._request('commit-chain', {
'message': message
})
def commit_entry(self, message):
return self._request('commit-entry', {
'message': message
})
...
The real meat and potatoes of this class is the _request method which essentially translates Python parameters into a format that the transport layer can deal with and asks the session to make a request. The result is either then returned or, if an error condition is present, sent off to the exception-handling code (below) to be turned into a native exception.
I prefer to explicitly define class methods for each API endpoint to add a little more structure, but you could just as easily do this in a generic way by accepting an RPC method name or URL as a parmeter, but then you're toeing the line of forcing the developer to consider server details.
exceptions.py
This module deals with (you guessed it) error handling. These are the bits that translate error conditions in API responses into useful exceptions the developer can work with.
To me, a useful API error has 3 components: Some kind of constant code or type that can be used in conditionals to vary behavior based on different categories of errors, a user-friendly error message explaining what went wrong, and additional metadata unique to the error type. To give you an example, here's what I consider a good form validation error:
{
"code": "validation_error",
"message": "One or more fields could not be validated.",
"data": {
"email": ["This field is required"],
}
}
From just this we know we're dealing with a validation_error type of error, so we should be looking for more data (in this case field-specific errors) under the data key to highlight problematic fields. We can also display the "One or more fields..." message in an alert to notify the user where to look.
Here's the error handling code:
# Take a HTTP response object and translate it into an Exception
# instance.
def handle_error_response(resp):
# Mapping of API response codes to exception classes
codes = {
-1: FactomAPIError,
-32700: ParseError,
-32600: InvalidRequest,
-32602: InvalidParams,
-32603: InternalError,
-32601: MethodNotFound,
-32011: RepeatedCommit,
-32008: BlockNotFound,
}
error = resp.json().get('error', {})
message = error.get('message')
code = error.get('code', -1)
data = error.get('data', {})
# Build the appropriate exception class with as much
# data as we can pull from the API response and raise
# it.
raise codes[code](message=message, code=code, data=data, response=resp)
class FactomAPIError(Exception):
response = None
data = {}
code = -1
message = "An unknown error occurred"
def __init__(self, message=None, code=None, data={}, response=None):
self.response = response
if message:
self.message = message
if code:
self.code = code
if data:
self.data = data
def __str__(self):
if self.code:
return '{}: {}'.format(self.code, self.message)
return self.message
# Specific exception classes
class ParseError(FactomAPIError):
pass
class InvalidRequest(FactomAPIError):
pass
...
The handle_error_response method attempts to build the appropriate exception class from the API's response including the error code, message, and additional data. If it can't do this then a generic FactomAPIError is raised with the default values.
Usually it's enough to catch a FactomAPIError, however if more control is needed we can catch specific error types, falling back to the generic exception:
try:
client.do_something()
except ParseError:
# Do something more specific here
except FactomAPIError:
# Do something more generic here
Other types of APIs
This pattern works well for RPC-style APIs, but tends to break down for more object-based or RESTful APIs as having a single interface class gets messy quickly. In those cases I find it makes more sense to break the interface down to resource-level, modeling things more like an ORM. I'll cover that in a later post, next time I find the need to build one.