Designing Great Web APIs (2015)
Chapter 2. Guidelines for Designing a Great API
If you have been involved with any kind of software product, you have probably heard of the term user experience, or UX. This is the discipline of meeting the exact needs of the user, from their interactions with the product, company, and its services.
Developer experience (DX) is just as important for APIs as UX is for great applications. The developer experience focuses on the various aspects of engagement with developers for your API product. This extends beyond the features of the API. It also includes all aspects of the API product, from first glance to day-to-day usage and support.
As we seek to understand how to design a great API, it is important to remember that DX is important for both internal and external developers. Internal developers benefit from great DX because they are able to create business value faster, especially for newer developers that just joined the organization. External developers benefit by integrating your well-designed API quickly, hopefully faster than any competitor can offer (or building it in-house).
As you begin to produce APIs for internal or external developers, you will be faced with a variety of decisions about your API. This can range from how you manage it as a product, to its design and documentation. The following seven guidelines will help your organization incorporate a great developer experience for the internal and external developers that will use your API.
#1 - Treat Your API as a Product
Many APIs start as a bolt-on solution to an existing application in an effort to solve a short-term problem, such as building a mobile application. However, APIs designed from the ground-up as a separate product enable the development of not just one type of application, but any number of applications that span a variety of devices and situations: third-party software, internal applications, desktop apps, mobile devices, and the emerging world of IoT. This means that APIs extend beyond a simple technology solution: they can have a positive (or negative) impact across the entire business, including:
1. Internal innovation
2. Marketing channels
3. Business development
4. Lead generation
5. User acquisition
6. Upsell opportunity
7. Device and mobile support
8. Increased customer retention
As a result, APIs need to be treated as a product, even if it is an internal product used only by the organization itself. This means investing in the API as you would any other product, including:
1. Clear communication of your API strategy and business model(s) to all lines of business to ensure everyone is focused on achieving the same business goals
2. Implementing an API governance program to encourage API consistency through design standards and product management
3. Managing the entire API software development life cycle (SDLC), from requirements to delivery, deployment, and customer support
4. Monitoring API usage and key metrics and performance indicators (KPIs)
5. Evangelizing the API to internal and external developers through online resources, hackathons, internal/external conferences, and partner programs
Taking these steps will ensure that not only will your API design remain well-designed and consistent, but also that your API will have longevity as you continue to support the needs of both internal and external developers.
#2 – Take an Outside-In Design Approach
Web APIs have a tendency to reflect two things: organizational structure and database structure. Neither will produce a great API design, and both of these situations can be prevented. Taking an outside-in approach to API design means looking at your API from the viewpoint of an external developer that doesn’t have access to your code, your database design, or your organizational structure.
Organizational structure leaks
When an organizational structure leaks into the API, it announces to developers that the API isn’t a product; it is the output of a specific team within the organization. The result is inconsistent design and duplication or gaps in functionality.
Organizational structure creeps into an API for two reasons: 1) APIs are designed in isolation from other teams, or 2) APIs are designed around internal systems rather than external needs.
To avoid this, consider establishing an API governance board that encourages consistency across teams. A governance board isn’t meant to be an oppressive committee that stalls innovation. Rather, they represent the best interests of developers, your APIs, and your business. Great governance boards often act as internal consultants for your APIs to make them successful.
Database structure leaks
Database structure creeps in through the use of tools and frameworks that promise rapid API development at the expense of a thoughtful API design. They externalize the data through generated APIs but make one huge assumption that becomes the enemy of great API design: external developers want to use your API like you access your database.
Developers using your API don’t care how you store your data as long as it is stored correctly and reliably. Taking an outside-in design approach will prevent the database design from creeping in by encouraging you to see your API as external developers will see it.
When taking an outside-in approach to API design, focus on how the API will be used, rather than how it is built. This means looking for ways to make your API useful for web clients, where connectivity and CPU capabilities are abundant as well as mobile devices, where connectivity and battery enforce a variety of limitations.
#3 – Write Great Documentation
The first essential trait of a great API is great documentation. It is important to remember that developers won’t have access to your source code. Therefore, they won’t be able to see how things work inside your API. They will depend on your API’s documentation to understand how to use it, when to use it, and what will happen as a result.
Great documentation requires careful attention to format, completeness, and discoverability. Let’s look at each of these documentation concerns in more detail.
Format
How you decide to distribute your documentation will heavily impact the audience. Many companies employ technical writers who use tools that produce beautiful, typeset documentation in PDF format. The documentation looks professional and amazing.
However, there is a big downside to PDF-based documentation: the PDF goes stale once it is downloaded. As new versions of the PDF are released, older versions remain on their hard drive for reference. This means that they won’t have the benefit of the latest examples, clarifications, and new features.
Instead, APIs need to have documentation that is always current for the reader. HTML-based documentation provides an always updated, easily accessible solution. It should be hosted on your website and easily accessed by developers to ensure the documentation is available at any time.
Completeness
At least one time in your life you have encountered bad documentation. It may have been the assembly instructions for do-it-yourself furniture. Perhaps it was an owner’s manual for a new device or software product. The experience of poor documentation can waste your time, leave you frustrated, or perhaps even cause you to replace it with something better.
API documentation can provide the same experience if it is left incomplete or unclear. Developers may continue to use the API in the short term, but they will likely look to replace your API at first chance.
It is important to remember that a public web API is a contract with every developer that will use it. Whether you target internal developers, developers within a partner organization, or public developers in general, you are making a contract with each individual developer.
Therefore, the documentation you provide to developers will provide the bulk of their developer experience. It will be the first thing that they experience when trying to understand your API, and the first place they go when they are stuck.
The challenge with API documentation is that it must serve a variety of situations. This may include the developer looking at your API for the first time or the expert developer looking to use a newly released feature. This means that complete documentation takes into account the following scenarios:
§ Developers, product managers, and business users that are considering your API but haven’t committed to it yet
§ The developer integrating your API for the first time
§ The expert developer who has been using your API for some time and wants to explore previously hidden features or options
§ Support staff trying to troubleshoot how your API works and why an application is failing
§ Newly hired developers in your organization seeing the API (and the business) details for the first time
Too often, documentation lacks focus around one or more of these areas. This is particularly the case when using documentation written inline with the code itself. Tools exist that allow API docs to be built from inline comments and are sometimes used to expedite the documentation process. However, the result is documentation that is only focused on what that specific code does, not how to use it successfully. Complete documentation must consider all of the five scenarios previously listed.
Interactive
Finally, great documentation should be interactive. As mentioned, API documentation delivered in HTML format can be hosted on a website and kept up-to-date. But another added benefit is that we can use the browser to actually interact with our API.
Figure 2-1 is an example of interactive documentation using Swagger.
Figure 2-1. API documentation using Swagger, which offers a Try It Out feature
Interactive API documentation allows anyone, including developers, quality assurance staff, product managers, and support staff to make API calls into a live running server from within the documentation itself. This is possible because our web APIs use the HTTP standard and therefore don’t require special libraries or software to make them work. Imagine allowing developers and QA staff explore your API before they ever write a line of code or an automated test! That is power of interactive documentation.
#4 – Have an Intuitive, Consistent Design
An intuitive API is one that makes it easy for a developer to know what to do to use it effectively. By lowering the learning curve of your API, you will ensure that developers experience successes early and often. This results in greater API adoption, longer retention (even in the face of competitors), and developer autonomy resulting in reduced support costs.
Follow these general guidelines to make your API more intuitive:
§ Make data available easily, rather than hidden or hard to find.
§ Require only the information necessary to accomplish the desired task.
§ Offer both low-level ways of getting things done, as well as higher-level ways of accomplishing common workflows with fewer calls.
§ Use hypermedia links to inform API clients the available actions at any given point, based on the current state of the data and the permissions of the API client.
§ Offer only one way to accomplish a task.
For APIs, consistency is important for a great developer experience because it creates predictability. As developers start to become familiar with your API, they will come to expect the same familiarity as they explore it further. This consistency encompasses a number of areas, including naming, resource URLs, payload formats, and error messaging. Let’s examine each one further.
Consistent naming
As developers approach your API, the first thing they will notice is the naming conventions that you use. This includes the names of the domain concepts referenced, many of which become the resources used in the URLs of your API. Therefore, naming is critical to the understanding and usage of your API. The following are some tips for incorporating consistent naming:
§ Avoid abbreviations, as they can be difficult to read and often create confusion as some names may be unclear or inconsistently abbreviated.
§ Be consistent with resource names to avoid confusion.
§ Refrain from referencing internal systems, as this results in requiring insider knowledge to use or comprehend your API.
Consistent resource URLs
As you design your API, you will need to map out the various URLs that will represent your resources. It is highly recommended that you develop a resource ontology to ensure that URLs are consistently designed into the API.
If you are unfamiliar, ontology is a technique used in information science for the naming and typing of entities and their relationships. API ontologies define the structure of your API, from the top-level resources to the nested resources under them.
Figure 2-2 is an example of a URL ontology for an ecommerce website.
Figure 2-2. An example resource ontology for an eCommerce API
When designing your resource ontology, apply the following techniques to create a more consistent API:
§ Use plural resource names when offering a collection of resources and a singular resource name for a single resource (e.g., /users for a collection of user resources, /user for a single user resource).
§ Use nested resources to indicate relationships between resources.
§ Avoid one-off URLs (e.g., /users/current rather than the more appropriate /user).
We will cover this topic in more detail in Chapter 4.
Consistent payload formats
Finally, the payload format should be consistent to allow API consumers to easily construct request payloads and parse response payloads. Most developers prefer to write helper code to handle this logic. If an API is inconsistent in its approach, developers will have to write one-off code to handle the differences. Follow these patterns for more consistent API payloads:
§ Reuse field names across payloads when possible for better predictability (e.g., use firstName and lastName or use fullName, but don’t interchange them).
§ Avoid abbreviations in field names.
§ Keep consistent casing rules, usually camelCase or snake_case (e.g., firstName or first_name, but not both).
§ Select a single payload format or standard when handling payloads, including resource representations and search result collections.
§ Ensure all error message formats are consistent across the API. This is especially important when an API is built by multiple developers who may otherwise handle error messaging differently.
Beyond data: Supporting API-based workflows
APIs often require more than just data. While data is important, developers seek out APIs to solve a problem. If they wanted to store data, they would push it into a database directly, unless you have data that no one else can provide. They are looking to your API to do something that they don’t want to build themselves. This may be solving difficult problems, building support for collaboration, or providing data analysis.
Great APIs need to look beyond data access endpoints. While these kinds of endpoints are essential to any API, adding workflow into your API helps developers get things done quickly and with fewer lines of code.
As an example, Twilio only supported conference calls by putting each caller on mute and then managing each caller using the low-level access APIs. The company realized that customers required an affordance to support conference calls at a higher level. The result is that providers were able to remove a lot of hard work from API developers by building in a higher-level conference call API that did the heavy lifting for them.
As an API provider, it is important to look for usage patterns of how your API may be used across a variety of different types of developers. Then begin to build in higher-level APIs that support these workflows, saving them effort and time.
A NOTE ABOUT HYPERMEDIA APIS
Hypermedia APIs are a breed of web APIs that inform clients about what actions are possible after a given request. This may take the form of next steps within a workflow, related API resources, and other functional areas of the API. This mimics how we use the Web: we go to a homepage, click the login link, enter our credentials, then navigate to various areas of the application.
Using hypermedia enables our clients to be more flexible and reduces the amount of business logic we have to code into them. Instead, they look for the existence of specific links within a response payload and offer (or hide) specific actions based on the server’s determination of what is and isn’t allowed. If the business logic needs to change in a workflow, the server simply adds or removes the appropriate hypermedia links that are used to guide the client’s behavior.
Some REST purists believe that hypermedia is a requirement for an API to be labeled REST. While this isn’t strictly the case, hypermedia often provides additional value to an API by making API consumption easier and more flexible. However, it isn’t necessarily required to deliver a useful and functionally complete API.
#5 – Design for Security at the Start
Teams can become so focused on the features that they forget to integrate security into their API design. Nearly every API will need to provide access to internal business systems, sensitive and personal information (PII), and/or public data that co-exists with internal/private data. Some APIs may provide limited data for free or for specific roles, while others may restrict access to any and all data. Therefore, every API must consider security from the design phase rather than as an afterthought, when it is often too late to make API changes without significant impact. The following are security considerations that need to be made during API design.
Authentication
Authentication is the process of verifying the identity of a particular API consumer. For websites that are built for humans, this is often a form that asks for a username and password. For APIs, there are a number of options available:
Password-based authentication
With password-based authentication, a username and password is sent on each request. While this may be secured through the use of transport-level security (TLS), software integrations to APIs may stop working if a user resets her password for any reason.
API-key based authentication
Instead of sharing a username and password, APIs can require the use of an API key that is provided by the service. This key identifies the API client and is not directly tied to a user’s password. The API key may be embedded within the URL as part of the request body or in the request header. API keys may be shared with other applications, but revoking access to the API key will cause all applications sharing the key to stop working.
Delegation-based authentication
For scenarios when third-party applications may want to connect to an API on someone else’s behalf (e.g., a third-party Twitter or Facebook client), delegation-based authentication is often the best approach. OAuth is the most popular standard, since it is somewhat like a “protocol of protocols” that allows disconnecting the authentication provider that provides API tokens from the API provider. API tokens are granted and revoked as desired by the user, allowing third-party applications to make API calls on their behalf (i.e., delegated access)
Authorization
Once you have identified yourself through the authentication mechanism, the API must still authorize access to the appropriate data and functional access rules for the system. This may be a direct result of your user account permissions within the API, or the permissions granted through the use of a delegation-based authentication mechanism such as OAuth. Each API endpoint’s authorization requirements should be considered during the API design process to ensure that data and access to critical systems are secured properly while the core functionality of the API is maintained.
Data leakage
Even with the proper authentication and authorization mechanism, your API design can still have security leaks. While this can happen for a variety of reasons, the most common is that APIs are designed for internal consumption only and are eventually promoted to partner or public developers. Sensitive data once thought as accessible only by internal systems is now made available to developers outside of the organization.
As an example, the Tinder API experienced a security breach through data leakage. While its mobile applications did not surface an individual’s exact location, the API did return specific locations within the response payload. This means that any developer had access to an individual’s location, since the data was not removed or scrubbed properly for external consumption. Instead, internal knowledge of another Tinder user’s location was both stored within their backend systems and then made available directly via its API, exposing these users to potential harm.
To prevent data leakage, design your APIs as if you were releasing the API to the public. This includes adding proper authorization for API consumers to grant or revoke access to specific data fields and functional areas of the API as appropriate.
Always use TLS
While some API endpoints may be providing data that isn’t sensitive, it is generally best to use TLS to secure all data in motion. This will help protect authentication credentials, as well as prevent eavesdropping when transmitting sensitive data between the API client and server.
Designing with security is important
As you likely realize by now, if security is an afterthought in your API design, everything from the API documentation to assumptions in implementation may need to be revisited. In addition, improper design of your API by ignoring security implications can result in the sharing of sensitive data and perhaps even exposing unnecessary risk to users.
#6 – Share Great Code Examples
Documentation is a very important element of the developer experience. However, code examples provide the important guidance necessary for developers to be able to apply the documentation in practice.
Code examples come in a variety of forms, from just a few lines that demonstrate how a specific API works, to more complex examples that show how to assemble multiple API calls into a complete workflow.
So, how do you choose what kind of examples to include? First, you need to understand the developer journey and the various milestones developers go through as they learn your API.
Milestone 1: First success
Initially, the developer needs to overcome any doubts that the API will solve his needs. This often starts with code examples that allow her to explore the API beyond the interactive documentation.
It is important to remember that during this phase, the developer just wants to see something work. This is often known as TTFHW, “Time to first Hello World” and is a key metric for determining API complexity. The longer it takes to get a developer to her first “win”, the more likely the developer will leave your API and find a better solution. If there are no alternatives for your API, the developer will build his own solution.
Products such as Twilio have made it their goal to onboard developers in less than five minutes. While this takes considerable focus and investment, your goal should be to find the shortest possible route, then keep working to shorten the time as your API matures.
To achieve a low TTFHW, provide concise examples that remove all need for boilerplate code. Look at the following example:
require "stripe"
Stripe.api_key = "sk_test_BQokikJOvBiI2HlWgH4olfQ2"
Stripe::Token.create(
:card => {
:number => "4242424242424242",
:exp_month => 6,
:exp_year => 2016,
:cvc => "314"
},
)
Notice in this example that there is little code to write. Simply fill-in your API key and the credit card credentials to try it out.
Bad example code requires that you write code to use the example. This requires you to learn more about the API before you can try it out. Never require developers to write code to complete an example when first trying out your API.
Milestone 2: Workflow support
After the developers have had some time to acquaint themselves with your API using your easy-to-use examples, the next step is to begin to demonstrate the API based on how they will want to use it.
Workflow examples focus more on achieving specific goals than applying coding best practices. This means that you want to convey clarity over code performance, clear intent over code quality. Use copious inline comments to explain why each step is necessary. Be willing to include hard-coded values for easier reading. Choose variable and method names that make it easy to read. Focus more on behavior than handling errors at this point.
It is important to note that while these examples will be more complex than those found from the first milestone, they shouldn’t exceed the height of the screen. The examples need to be short enough to explain the concepts but not too long that they require considerable time to understand. It is often best to demonstrate scenarios that are easily understood and likely map to your customer needs.
Milestone 3: Production-ready integration
Once the developer has followed your simple examples and then tried some workflows, the final step is to help her understand how to integrate your API into her production environment. This includes how to catch errors to help developers properly troubleshoot their integration code. It also includes demonstrating how to catch and recover from bad data provided by end users. Finally, if you are enforcing rate limiting, then show not only how to obtain the current rate limits for their account, but also provide tips for reducing the number of API calls required.
#7 – Provide Helper Libraries
Web APIs have a huge advantage: they are built on the hard work of the authors of the HTTP specification. This means that any developer can pick up their favorite programming language, select an HTTP client library, and make API calls. They don’t need anything else to make it happen.
Helper libraries allow developers to make API calls using a special library built for their specific programming language. As an example, a Ruby helper library can be provided to allow developers to consume your API directly, without having to write the code necessary to handle the lower-level HTTP logic required for web APIs.
While API providers are not required to release helper libraries, they often accelerate developer success. This is especially the case for mobile developers, who are accustomed to working with helper libraries rather than raw HTTP. Many API providers that wish to reach mobile developers choose to produce a helper library for iOS and a helper for Android.
For APIs that will likely be used outside of a mobile platform, you may opt to provide helper libraries for Java, JavaScript, Python, Ruby, Go, PHP, and .NET. What languages you choose depend upon your intended audience. For example, if your target audience is enterprise developers, then Java and .NET should be priority. For web developers, you may choose to focus on Python, PHP, JavaScript, Ruby, and Go. Whatever you do, don’t assume that your team’s preferred language platform is what your customers will be using. Do the research to be sure you have prioritized your focus.
It is important to note that each helper library should be documented fully and follow each language’s programming idioms. This will prevent a library written in Ruby from looking like it was built for a Java developer. This may require hiring outside expertise to consult on how the library should work, or perhaps even build it for your organization.