Darrel Miller on whether Hypermedia APIs will save the universe
/
Developer evangelist and developer Darrel Miller has more than 20 years' experience building distributed business applications and is recognized as an industry expert on the use of HTTP, REST and Hypermedia to build connected systems.
The Hypermedia API is certainly a controversial topic, regaining traction as it is claimed to be a solution to the Internet of Things. This more accepted nickname for HATEOAS [Hypermedia as the Engine of Application State], has been around for awhile now, mainly in the HTTP space, but is making headlines again with new potential use cases. To jumpstart The API Economist again, we thought we’d start closer to home with one of our own Azure API Management team members Darrel Miller, co-host of YouTube channel “In the Mood for HTTP” and owner of the Bizcoder blog. Darrel is a veteran software developer with more than 20 years of experience in all things HTTP, Hypermedia, HttpClient and the Web API.
The API Economist: How about we start by you telling us a little bit about your background in the API economy.
Darrel Miller: I found my way into the API economy via my own path back in 2005. I was building a large “line of business” application for small manufacturing businesses. I built some APIs [application programming interface] using SOAP and suffered due to tight coupling with client code. Minor changes on the server would cause clients to break. I made a conscious decision to learn how to use HTTP as a true application protocol and take advantage of its natural capabilities. I went on to design a large Hypermedia HTTP API that drives numerous client applications. Since then I have been an active writer, speaker and OSS contributor in the world of HTTP and APIs.
What is a Hypermedia API?
A Hypermedia API is one that returns links in the HTTP responses that a client application follows to achieve defined goals. Which link the client follows is based either on client-based algorithms or human input. Many people make the distinction between links and forms, I don't usually bother. A form is a link with parameters that need to be filled in. If you want to sound clever you can call them affordances.
What is the benefit of a Hypermedia API?
A Hypermedia API allows more of the application logic to reside on the server. This allows business logic to change without having to deploy new clients. Also, by using links to reference resources, the location of resources can change without breaking clients. This enables systems to scale without client redeployment.
What are some sample use cases?
One of the easiest scenarios to envision is the implementation of a kiosk in some retail or commercial location. Consider the automated check-in terminals at an airport. The user interface must guide the user through a series of selection screens that capture information and then ask additional questions based on responses. The logic needed to decode what questions to ask next could be implemented in the client application, however that would mean doing an upgrade to the client software whenever that logic changes. Some kiosks provide check-in services for multiple airlines. Who manages the updates? Because hypermedia-driven clients can be ignorant of resource location, a client can consume services from multiple servers simultaneously and make those appear as a single service. The links in responses can guide a client through the data collection process and logic on the server can ensure that only links that are relevant based on previous responses will be included.
Tell us about a specific situation where a Hypermedia API was a solution for you.
In a previous life, I built line of business software for small metal fabricating companies. Over 20 years, I built three versions of this software to manage all the operational tasks within a company. The first was a DOS file server-based application, the second a Windows Client/Server application and the third a .NET hypermedia-driven system. There were several client applications and a very large Hypermedia API. Deploying client components to many differing types of hardware in a small company that does not have a sophisticated IT capability can be challenging. The use of hypermedia to enable loose coupling between client applications and APIs allows for a more flexible approach to updating system software components.
What companies are doing just the right thing in using Hypermedia APIs?
The only companies that I have seen to date that appear to have fully embraced hypermedia for a public API are Huddle and Foxycart. However, there are an increasing number of companies who are including links in responses to enable navigation between related entities. Examples include Github, Stormpath, and Runscope.
I think it is important to make a distinction between hypermedia that is used to passively present relationships between resources and APIs that use links to drive the user experience. Using links to relate data decouples the client from knowledge of resource location and API URi design conventions [uniform resource identifier]. This has some benefits but doesn't really allow much application logic to be removed from the client.
In any given situation, how much harder is it to create a normal API vs a Hypermedia API — what are the trade-offs? Hypermedia specific obstacles?
To add passive links to API responses, it is probably a 10 percent increase in effort to produce a hypermedia-connected API. You need to select a format to render links. You need to find a way of resolving URLs to embed. This can be more tricky when your API is behind reverse proxies and load balancers. You need to make decisions about relative versus absolute URLs.
To build a hypermedia-driven API is a very different challenge than building an API that exposes data or discrete functions. It is hard to compare the two because they are approaching the problem in a very different way and a hypermedia driven application requires a significantly different approach to building a client application, further making the comparison challenging.
Why are most APIs then not hypermedia?
The benefits of passively connected hypermedia are not usually significant enough to convince developers to do things differently. Many of the benefits are only seen over the long term but the cost is paid up front. For hypermedia driven applications, there is a tricky catch-22 situation. I believe it is difficult for someone to build a good hypermedia driven API without having experience consuming one. Unfortunately due to the lack of publicly available hypermedia driven APIs it is difficult to gain this experience. I was fortunate enough to have five years, minimal deadlines and the ability to do it wrong many, many times in order to realize what is possible. Very few people have this luxury.
Why aren’t there more vendors creating tooling for these frameworks?
Hypermedia driven applications are a style of building application. Once you have media types that can hold links, and library that can make HTTP requests, you don’t need much more tooling. There is a significant opportunity for vendors to build reusable components that can take advantage of hypermedia, but my experience has been that they just have not seen the possibilities yet.
Are Hypermedia APIs just universally great or are there just better situations? Really, when do they work better?
Hypermedia APIs work best for driving client applications through well defined scenarios. They are not nearly as well-suited for open-ended APIs that are simply exposing data. Hypermedia APIs are great for building very thin native clients across multiple platforms that should provide a common experience. Hypermedia APIs are useful when deploying client updates is difficult. Hypermedia APIs are not particularly useful when client and server are sitting in the same data center, managed by the same team and can be deployed simultaneously.
What about OData?
OData mostly falls under the "passive links" style of hypermedia. It also defines URL conventions that make it possible for clients to generate URLs and therefore embedding links is somewhat redundant. OData does support the notion of Action links but they are not commonly used from what I have seen.
I usually find it much harder to convince hypermedia skeptics of the value when only discussing relation links. It the active links that guide clients through a workflow that provide a much stronger argument.
Do you consider yourself a hypermedia purist or a hypermedia pragmatist? Is there a middle ground for HAPIs? Do you have to go all-in on Hypermedia to get the benefits?
This is one of those distinctions that annoys me. In the field of software engineering we are in the business of making trade offs. Every design choice should be made by weighing costs versus benefits. Frequently I find the term pragmatist is used to dismiss certain approaches because the costs versus benefits are not fully understood.
Hypermedia is just one of many approaches that can be used to manage client/server coupling. If can be used to many varying degrees. In fact almost all HTTP APIs already use it in some ways. Whenever a API has some kind of image, be it a photo or an avatar, it is almost always pointed to with a URL. This just makes sense because it is so inefficient to embed an image into a JSON representation and images are frequently stored on different servers.
There are many ways that APIs can begin to incorporate more hypermedia into their APIs. However, I don’t recommend anyone does so until they understand the benefits that it will bring to them.
Darrel API evangelizing at a recent APIDays conference.
How have you used Hypermedia APIs for HTML and adaptive web design?
Web browsers use Hypermedia APIs all the time. This is one of the frustrating parts of discussing hypermedia-based systems. The human Web is a massively successful example of a hypermedia-based system, but, unfortunately, through the use of JavaScript, we found many non-hypermedia-based solutions to the limitations of HTML to do the vast variety of things that we do with it. This makes the conversation about hypermedia in a web browser fairly confusing. It is possible to build hypermedia-based systems using JavaScript but you need to recognize that you are building a hypermedia system inside a hypermedia system. It is fairly mindboggling.
Hypermedia is mostly orthogonal to adaptive web design. The architecture of the Web makes a distinction between a resource and representation. The resource is the concept that is of interest to the client, the representation is a physical—well as physical as bytes on the wire can be—manifestation of that resource that is appropriate for the client device. Hypermedia points to the resource, it is up to the HTTP mechanisms to ensure that the most suitable representation is returned.
What languages have you used with Hypermedia APIs?
I have used C#, Go, Python and VB6 to work with Hypermedia APIs, however, it really makes no difference what language is being used as long as the language has libraries that allow fetching a response from a URL and interpreting that response.
Have you tried JSON-LD and HYDRA?
I have done a small amount of work with JSON-LD and I have attempted to read the HYDRA spec a few times.
JSON-LD is definitely the best attempt to date to make RDF [Resource Description Framework] consumable by mere mortals. However, it still suffers from the RDF problem—it is too clever. I believe there is a sweet spot where JSON-LD can be used to apply application semantics to real problems. Getting to the point where you are comfortable with the RDF triple model is challenging and those who achieve that level of enlightenment quickly get sucked towards the holy grail of the one true data model. In doing so, the solutions they produce are that much less approachable to those who are not enlightened. The sophistication limits adoption which is the one thing that is truly necessary for interoperability. Many of the XML-related specifications suffered the same fate and eventually XML succumbed to JSON due its "simplicity" despite the fact that you can also use XML is an equivalently similar way. Now as JSON gains maturity, the previously shunned complexity is being added back into JSON by a new generation of developers.
I am not convinced of the value of HYDRA. Its mechanisms to augment representations with links seem redundant to me and I don't see the value of yet another API description language that encourages use of HTTP in tightly coupled way.
Tell us about the difference between human-to-machine interaction and machine-to-machine interaction with Hypermedia APIs?
There is no difference. I realize that most API experts will disagree with me, but I believe it is not helpful to view this a binary characteristic. I believe there are only M2M [machine-to-machine] interactions with varying degrees of human involvement. A human being cannot make an HTTP request. A human can interact with a computer, that is running client software, whether it is a web browser or a native application, that can make a HTTP request.
It is common for people to perceive things like static websites as H2M [human-to-machine] interactions. However, static HTML pages are just HTTP API responses that are interpreted by a client application. A significant portion of API interactions made by web browsers directly correlate to human interactions. However, even in static websites, there are plenty of algorithmic driven API requests. Accessing stylesheets, images and JavaScript libraries are all API requests made by the client to be able to render the UI [user interface].
At the opposite end of the spectrum, some HTTP clients, like web crawlers have very little human interaction. They are probably configured and scheduled for execution by a human, but beyond that, they run based on an algorithm designed to find and follow links.
Is there any tooling out there to help use Hypermedia APIs?
There are numerous hypermedia media type formats that are now in common use. Formats such as HAL, Collection+JSON, Siren, Mason, JSON API, Uber, JSON-LD and OData all provide hypermedia support.
For a number of years, I have been building tooling in the .NET space that is designed to make it easier to work with hypermedia. My Github repositories for that stuff are at http://github.com/tavis-software and http://github.com/hapikit. There are many other tools that people are working on but rather than me attempting to list them and missing many of them, I would suggest joining the Hypermedia-Web Google Group and ask the question there what tools are available for your preferred platform.
How do Hypermedia APIs factor into the Internet of Things?
Hypermedia's ability to reduce the need to update client code is very useful in the IoT [Internet of Things] world because updating the code on IoT devices can sometimes be very difficult. However, hypermedia's usefulness is fairly dependent on having the ability to connect to the server. For devices that are regularly offline, using hypermedia may introduce additional challenges.
What will the future of Hypermedia APIs look like?
If Hypermedia APIs other than websites ever do achieve critical mass, I don't believe they will be radically different than those few that exist today. However, I think the way people will build clients that consume APIs will be massively different. The possibility to build "many-to-many" client-server relationships with standardized media-types, link relations and semantic vocabularies has the potential to radically change the design and capabilities of client applications.
Do you have any tips of the trade for using Hypermedia APIs?
Make more use of link-relation types. Don't be afraid of creating your own, but try and create as few as possible. If that sounds contradictory—welcome to the world of engineering [winking].
What is the easiest way to get started building hypermedia APIs?
One suggestion that I regularly make to developers who want to dip their toe in the world of hypermedia, is to try creating a discovery document at the root of their API. Something like Github do at their API root https://api.github.com/. Many people when building APIs don’t actually put a resource at the root of their API and so adding a root document rarely interferes with the existing API. The root discovery document can be used optionally by client developers to completely decouple their client application from hardcoded URLs on the client. It can also be used a simple inline developer API documentation.
What will the API economy look like in five years?
I am hoping that within five years most companies have learned their lessons from the first attempts at producing APIs and are building APIs that will not break consumers regularly and deliver value based on well understood and economically viable scenarios. Many companies are still rushing into API development with the naïve perspective of, ‘Let's expose our data and see what things people do with it.’
What should a developer be doing to prepare for the future of the API economy?
That's easy. Make sure you have a thorough understanding of how HTTP is designed to work. [smirking]
What are your three favorite apps? (Not the ones you like the most, the ones you really use the most.)
Who is the must-follow, undercover API expert on Twitter?
Mark Baker @distobj is the one person that doesn't get the recognition he deserves for the impact he has had on the Web and the evangelizing of REST. Although, Mark keeps fairly quiet these days, he has plenty of history of writing on the Web that is filled with invaluable wisdom.
Do YOU have more questions for Darrel on Hypermedia APIs? Or perhaps you’d like to chime in? Comment below and share your thoughts to @APIEconomist and @Darrel_Miller!
Photo courtesy of APIDays.
