Functional requirements of tool for REST testing

In this blogpost, I am going to propose functional requirements of tool which should test REST constraints of some API. Do you think there are some more? Feel free to share them in comments.

1. Inserting an API entry point
   Application will provide a form for specifying an API entry point. This form will consist of following fields:
   • text field for API's entry point URL address
   • a field for selecting an HTTP verb to use on that address
   • optional fields for specifying custom headers
   • optional field for specifying request body
   After the form is filled, the user can execute runtime testing by clicking the send button.
2. Runtime asking
   During runtime testing, the application traversing through API from specified entry point using Uniform Interface. However it might be required to modify some of automatically generated requests. For that purpose there is runtime asking form which includes following fields:
   • exclusion button for exclusion this request from runtime testing
   • optional fields for specifying custom headers
   • optional field for specifying request body
   After the desired inputs are filled, the user can save the modified request by save button.
3. Displaying of runtime testing results
   The application is displaying discovered resources in tree structure with informations about REST constraints fulfillment such as cacheability and auto discovery capabilities. The tree displaying should be optimized for viewing at mobile devices as well as on computer screens.
   Every resource in tree should allow editing of its request details by Runtime asking form and should be re-runnable manually.
4. Saving state and result of runtime testing
   The application can save and export current state of runtime testing and currently obtained results.
5. Restoring state and result of runtime testing
   The application can restore or import state of runtime testing and obtained results and allow its continuing.
6. Response correctness checking inputing
   The application allows user to manually enter expected response of particular request. The user do this by filling a form which contains:
   • text field for resource's URL address
   • a field for selecting an HTTP verb to use on that address
   • optional fields for specifying custom headers
   • optional field for specifying request body
   • area for specifying expected response code, headers, body
   After the form is filled, the user can save that expected response by save button.
7. Response correctness checking generating
   The application allows user to generate a set of expected responses to requests from API documentation. The user do this by specifying API's documentation path.
8. Response correctness checking evaluating
   The user can run and review previously inputed (generated) response correctness in table form. Each row for one request/response test with information about if passed or not passed.

This above would be my wish-list about functional requirements for REST API testing application. There is much likely more what should it do, for example I omitted stress testing in this stage of development. If you have any comments, suggestions just let me know.

Which applications can be used to test RESTful API?

In this blog-post, I am going to look at some existing tools suitable for testing REST API. I'll try to evaluate how they do tests and what they are testing and how it can be used for testing REST API.
These tool are: soapUI, Apache JMeter, FitNesse, Fiddler 2 and JUNITs + maven build + Spring MVC REST.
I am not going to list all of the features the tools poses, just those interesting for REST testing.

soapUI 
( http://www.soapui.org/ )
licence: open source,
platform: Java,
available as: standalone app or as plugin to IntelliJ, Eclipse, NetBeans, maven.
Through graphic interface, soapUI allows you to easily and rapidly create and execute automated functional, regression, compliance, and load tests. Key features are:

• Can test SOAP- and REST-based Web services, JMS enterprise messaging layers, databases, Rich Internet Applications etc.
• It can mock services (no automatic generation for REST).
• Can record, analyze and even modify client-server traffic as it happens.
• Offering Drag and Drop Test Creation.
• Nice request builder.
• Nice manual.

There is option to buy pro version with aditional features such as Groovy Scripting, Data-Driven Testing.

Typical use is to manually create a TestCase - a set of specified HTTP requests and expected responses which soapUI evaluates.


Apache JMeter 
( http://jmeter.apache.org/ )
licence: open source,
platform: Java,
available as: standalone app.
Java application designed to load test functional behavior and measure performance. Is usable through graphic interface or command line. Key features are:

• It can make a graphical analysis of performance or test your server/script/object behavior under heavy concurrent load. Supports regression tests.
• Can test HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS Mail.

Isn't focused on REST testing but since its capability to send and evaluate http requests we can use it this way - create a test case like with soapUI.


FitNesse 
( http://fitnesse.org/ )
licence: open source,
platform: Java,
available as: standalone app.
FitNesse enables customers, testers, and programmers to learn what their software should do, and to automatically compare that to what it actually does do. It compares customers' expectations to actual results.
You can Collaboratively define Acceptance Tests and execute them.

Typical use is to define tests by wiki pages. Then wiki pages uses binding (Java) code to execute methods of tested app. The binding code is written by you and is meant for making calls to jars of tested app. Because of that, you will have to tveak your binding code in order to call REST API and I don't think that FitNesse was designed this way.


Fiddler 2 
( http://www.fiddler2.com/fiddler2/ )
licence: freeware
platform: .NET,
available as: standalone app.
Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect traffic, set breakpoints, and "fiddle" (read, create, modify http responses and requests) with incoming or outgoing data. You can do this by graphic interface.


JUNITs + maven build + Spring MVC REST
This is not out of the box app for testing. It's 3 separate products which can be used together to help you create test application for REST API.

JUNIT is a simple framework to write repeatable tests for the Java programming language. It is an instance of the xUnit architecture for unit testing frameworks. Alternative to JUnit can be TestNG.

Maven build refers to Apache Maven. Maven can manage a project's build, reporting and documentation from a central piece of information.

Spring MVC REST refers to REST support in Spring's MVC framework. Spring is modular application development framework for enterprise Java. It allows you to focus on your business problem rather than the plumbing code that connects components and systems. Spring MVC REST allows you to annotate java methods (that contains just business logic) which are supposed to communicate using REST. The framework then generates plumbing code for these methods and add it to you application.
Alternative to Spring MVC REST can be Jersey.

By using these three tools you gain the big advantage of writing your testing app completely suited for your needs. That's of course the biggest disadvantage too as you get absolutely no functionality out of the box.

Summary
So that was some apps for testing. Except JUNITs + maven build + Spring MVC REST which is too general, they all have one thing in common. They just inspect and evaluate HTTP requests/responses and compare them against expected values you entered. They doesn't lead you to create truly RESTful API, nor they test if it is really RESTful.

We can ask ourselves if we can combine these apps to solve more then what single app offer. Well only particularly. SoapUI has great GUI for creating request and view responses. JMeter is shifted more to performance testing. FitNesse offers wiki based UI for inserting tests. Fiddler can control packet flows between your app and world. But even if we combine them, we still just get evaluation of response against expected response. We are still not able to test REST constraints.

So if we are going to create new testing app. Based on all these findings we can say we don't need another request/response inspector. We have to focus on testing REST constraints compliance and let other tools do the "expected response evaluation".

Is it possible to automatically test REST constraints?

In this post, I am going to look at testability of REST constraints. What interests me most is if we can test them one by one automatically? Are we able to write bot, program or some test framework which can tell "yes, your api is/isn't RESTful"? I am going to focus on REST constraints implemented on HTTP protocol.

You can take look at the constraints of REST and what benefits they brings at my previous post.

Constraints we can test automatically:

Cache constraint can be checked by looking at HTTP headers Cache-Control, Expires, Last-Modified. For fulfillment of this constraint, every response from server should contain them.
There is actually an online tool - redbot which analyzes the headers and tells about cacheability (among other things).


Constraints we maybe can test automatically:

Client-Server - HTTP itself is client-server. But we should somehow test if server just listens and wait for requests. My proposal: if we obtain a response after our client send request is it satisfying to tell this constraint is met?

Uniform Interface - my proposal: if this condition is met, the generic client should be able to go through all server's services (from one starting point) and used them without any special client's implementation. The client should only use standardized methods with communicating with server. 


Constraints we don't know how to test automatically:

Null style - we would have to test if some of "all other constraint" isn't present in the app. That's pretty huge number of tests ;-)

Stateless -  the programmer of the server can probably tell if it is stateless but if the server is black box for us, how to test if it keeps some state? Perhaps we can send same requests in a row and assume we always get same answer?

Layered System - the programmer of the server can probably tell if his system is layered, but how can we test it if server is black box for us?

Code-On-Demand - just optional constraint and again, the server programmer can tell if the server is capable of that.


Conclusion:

We can test one condition. We maybe can test another two. We cannot test another four but three of that four can be answered by programmer of server.

REST constraints and their impact on web application

What can be gained by making the api of our web application RESTful? What are the constraints of REST and what advantages can our web application loose when we decide not to follow some of REST constraints?
The answers to these questions lies in Roy's T. Fielding dissertation so I'll just list them here.

Null style: means there are no other constraints then following.

Client-Server: client wants something, server has it and can be asked for it by client. Still client and server can evolve independently.
Fulfillment of Client-Server constraint improves portability of the user interface across multiple platforms and improve scalability by simplifying the server components. And components can evolve independently.

Stateless: the request from client to server must contain all information so the server can understand the request. In other words, only client keeps information about it's state.
Fulfillment of stateless constraint gives our applications following three traits:

• Visibility - monitoring system does not have to look beyond a single request in order to determine the full nature of the request.
• Reliability - eases the task of recovering from partial failures.
• Scalability - server can quickly free resources and doesn't have to manage resource usage across requests.

Cache: all the data within response can be labeled as cacheable or non-cacheable.
Fulfillment of cache constraint is improving network efficiency, scalability and user perceived performance by reducing the average latency of interactions.

Uniform Interface: implementations are decoupled from the services they provide. Informations are transfered in standardized form. To obtain an uniform interface we need to define interface constrains. For REST they are:

• identification of resources: we can address desired information and we can recognize them.
• Manipulation of resources through representations: we are manipulating with representation of resource, not with resource itself and in a format which suits the client. Although it can be same as original resource.
• Selfdescriptive messages: the messages consist of data, metadata describing the data, and optionally metadata describing metadata.
• Hypermedia as the engine of application state: server is master of it's resources. The resource implementation details are hidden behind the interface. Server publish it's interfaces as resource hiearchy. Server does not tell which method is supposed to use with which URI.

Fulfillment of Uniform Interface constraint simplifies system architecture and improving visibility of interactions. It ensures the independent evolvalibility because implementations are decoupled from services they provide

Layered System: whole system is divided to layers. The component of each layer can "see" only it's immediate layer.
Fulfillment of Layered System constraint improves internet-scale requirements. Bounding overall system complexity and promoting substrate independence.

Code-On-Demand (optional constraint): client functionality can be extended by downloading and executing code (scripts for example).
Needed features doesn't have to be implemented in client at the time of compilation.

REST maturity - 3 levels

Approaching the REST API implementation, there are some inaccuracies among programmers community. Some of them publish an API called REST which is in fact not REST because it violates some of it's constrains.

Leonard Richardson discussed "The Maturity Heuristic" at 2008 QCon speech. This can be viewed as levels of REST, or maturity levels of REST, or simply maturity of REST. I'd like to present my summary of his speech:

Level 0:
One URI and one HTTP method:
client can communicate with web app only through one URI, using only one method. This is like SOAP. We are using HTTP protocol here only as a container for our messages which carrying all the control information in the HTTP body.

Level 1:
Many URIs and one HTTP method:
it is level 0 with extended possibility - address concrete things. It's giving us opportunity to use only part of such web app (by addressing concrete thing through its URI) if we want. But we still have to put control information into HTTP body if we need further actions over URI.

Level 2:
Many URIs, each supporting multiple HTTP methods:
we can address desired resource and we can tell what to do with that resource by using HTTP verbs only. In the body there are no control information allowed. It can contain only content for addressed entity.

Level 3:
Many URIs, multiple HTTP methods and resources describing their own capabilities and interconnections:
if you enter some resource, it not only tells you what can you do with it. It tells you what are the actual URIs of possible actions, even what are the actual URIs of relevant elements(other URIs) and their actions. This is something called HATEOAS.
Best thing about level 3 is that you don't need manual to use it or to write client for that. You actually doesn't need to write client which is specific for some webapp because generic client can use all webapps which are at level 3.

So which of these mentioned levels belongs in REST?
The original author of REST, Roy T. Fielding clarifies common mistakes about REST API on his blog. He is very restrictive about what is REST and what's not - no levels allowed. If we apply his rules to levels above we can tell that only level 3 is REST. The other levels are not. So API which is below level 3 are not RESTful.

Summary of some publicly available REST APIs

Below is list of some publicly available REST APIs (or APIs close to REST). I am going to take a further look at them. Any suggestion for another interesting REST API is welcome.

• API for Amazon S3 (Amazon Simple Storage Service),
• API for Google Cloud Storage,
• API for PHPRESTSQL (A HTTP REST interface to MySQL written in PHP),
• API for Dropbox,
• API for bitly,
• API for goo.gl,
• API for Myspace,
• API for Netflix,
• API for Twitter,
• API for SalesForce,
• API for delta cloud,

Thats all for now. The ordering in list is random.

Motivation for programing the RADREST framework

The server sides of today's Internet applications are very similar. Of course, every app is different by it’s functionality and complexity. They typically follow the MVC pattern using some existing framework providing desired business functionality. The applications contains a lot of similar source code. It runs on own server and it’s hard to use by other web apps. The development and maintenance must be made by experienced programmer.

What is typically the purpose of such applications? It’s processing data! More formally we can say that it’s purpose is exchanging representation of information. Do we really need to do so much server side programming? Why should programmers repeat the handlers coding instead they should focus on desired functionality? The only thing we need to know is where on the web is the needed information. If we knew where to find it, we can easily read it or modify it. Our believe is that we can design desired business functionality putting together couple of REST services. The resources can be ours or provided by 3rd party companies. We also want open the door to non-programmers providing simple server side REST assembly environment.

Our approach offers several advantages for web applications designers. The end users would be able to easily develop and publish the desired business functionality without deeper programming knowledge. Programmers on the other hand would be able to assembly even very complex applications really fast and easy. And maintenance of such apps will be also simplified. There won’t be any server side programing. As for apps, there are also several advantages. For example, moving apps from server to server wouldn't be any problem anymore. Creation and deployment of an app will be matter of few clicks. An app that obeys REST will be easily scalable.

The user owned REST services will be provided to third parties applications. They will serve external application, without knowing the implementation details. This is opening new business with the REST based services. Big corporations can create REST services exchange or replace dozens of Informational Systems they already used.

These are the reasons why we believe we need to move from object oriented programing (OOP) to resource oriented programming (ROP) and facilitate it’s easy deployment on server.