April 26th, 2011

An Extensible Rest API

I have the ideas in this post since some time, but I haven’t decided to publish them because they have not really been tested in any project of some complexity and therefore have not passed the theory level. Anyway, after reading the article “HTTP-CQRS: REST + RPC” I have decided to write this, because I think they respond to the same problem.

That paper argues that Rest is good as a system for querys on information, but it doesn’t works well as a system to make edits or actions on these documents. This is due to the “symmetric data models”:

The query and write data models Are Treated as symmetric, When in reality What we query for and What We manipulate seldomly Follows the Same model.

(read the article for a very clear explanation about the problem to which we refer).

I totally agree with the statement of the problem, but not with the solution (to use HTTP-RPC procedures for writing and Rest for querys).

My proposal is to do what in Spanish is said “to make a virtue of necessity” and from a possible limitation construct a more flexible system. We propose to follow this rules:

  • Separation of information (the original data)  and functionality (results / calculations): documents stored by some client applications should not be altered by any service, they should be considered the original source of the data. No service should consider himself as master of the original data upon which it operates. If I (my client application) have decided to save a document with some information, when applying for the content of the document it must contain this information, without additions or arbitrary modifications.
  • The application functionality is provided by external services that are notified of the changes occurring in the source documents and can maintain updated his “calculated” information.  It is therefore important that the service responsible for storing the source documents have some notification  system.

A clear example of this would be a service to save certain types of documents with a word search functionality. There must be an external “indexing” service who receives notifications of original document modifications and just index them (by the way, this is exactly what Elastic Search does in conjunction with couchDB with its “The River” API).

Another example could be a billing system:

There is some generic “Store” service for storing documents with the information for each customer and a document for each invoice. There is also a “Billing” service witch understands the content of these documents and give informations such as customer data with its billing digest, monthly reports, etc. If, for example we decide to add category information to the bills, we can create another independent service witch  “understand” that category information and give us their abstracts.

Note that this system is perfectly suited to document-oriented databases, in the NoSQL style.

The main problem that I see with this scheme is that the various services must be able to have some validation system for the modifications of source documents. For example, we cannot be allowed to delete a customer if it has bills.

Here are some possible solutions, depending on the degree of your extensibility requirements:

  • the store service allows others services to subscribe to a validation service
  • following the CouchDB model, each service could indicate conflicts to be resolved in the editions
  • more direct: the store service is not accessed directly, but only via the “Billing” service that validates each document before sending it to the storage service

As I said this is still in a theoretical level and therefore its effectiveness is to be evaluated. Admittedly, it adds a degree of complexity to the design of services, but in the other side it is based on some common, standard services, so once implemented basic storage services, notifications, etc. each specialized service  can concentrate on their specific functionality.

Note I. Extensibility:

Extensibility in Object Oriented Frameworks are generally based on interfaces. One of the characteristics of Rest is its “uniform interface”. This makes Rest is inherently extensible and extensibility considerations have to be of another higher order.
As the interface is fixed, all issues on extensibility must be about the organization of the content of resources (some peoples understanding of Rest could also consider the organization of the URL as part of the design of an Rest api, but our point of view is that the api has to be independent of URLs). In this article I have draft an resource content organization that allows different, independent services to act over the same resources adding new functionalities to them.

Note II: and what about the “CQRS (Command-query Resposibility Separation)”?, well I’m just learning about it… My first thought is that it is more traditional SQL oriented and my approach is more adapted to document databases.

As always any ideas and criticisms are welcome.