Which applications can be used to test RESTful API?

In this blog-post, I am going to look at some existing tools suitable for testing REST API. I'll try to evaluate how they do tests and what they are testing and how it can be used for testing REST API.
These tool are: soapUI, Apache JMeter, FitNesse, Fiddler 2 and JUNITs + maven build + Spring MVC REST.
I am not going to list all of the features the tools poses, just those interesting for REST testing.

soapUI 
( http://www.soapui.org/ )
licence: open source,
platform: Java,
available as: standalone app or as plugin to IntelliJ, Eclipse, NetBeans, maven.
Through graphic interface, soapUI allows you to easily and rapidly create and execute automated functional, regression, compliance, and load tests. Key features are:

• Can test SOAP- and REST-based Web services, JMS enterprise messaging layers, databases, Rich Internet Applications etc.
• It can mock services (no automatic generation for REST).
• Can record, analyze and even modify client-server traffic as it happens.
• Offering Drag and Drop Test Creation.
• Nice request builder.
• Nice manual.

There is option to buy pro version with aditional features such as Groovy Scripting, Data-Driven Testing.

Typical use is to manually create a TestCase - a set of specified HTTP requests and expected responses which soapUI evaluates.


Apache JMeter 
( http://jmeter.apache.org/ )
licence: open source,
platform: Java,
available as: standalone app.
Java application designed to load test functional behavior and measure performance. Is usable through graphic interface or command line. Key features are:

• It can make a graphical analysis of performance or test your server/script/object behavior under heavy concurrent load. Supports regression tests.
• Can test HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS Mail.

Isn't focused on REST testing but since its capability to send and evaluate http requests we can use it this way - create a test case like with soapUI.


FitNesse 
( http://fitnesse.org/ )
licence: open source,
platform: Java,
available as: standalone app.
FitNesse enables customers, testers, and programmers to learn what their software should do, and to automatically compare that to what it actually does do. It compares customers' expectations to actual results.
You can Collaboratively define Acceptance Tests and execute them.

Typical use is to define tests by wiki pages. Then wiki pages uses binding (Java) code to execute methods of tested app. The binding code is written by you and is meant for making calls to jars of tested app. Because of that, you will have to tveak your binding code in order to call REST API and I don't think that FitNesse was designed this way.


Fiddler 2 
( http://www.fiddler2.com/fiddler2/ )
licence: freeware
platform: .NET,
available as: standalone app.
Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect traffic, set breakpoints, and "fiddle" (read, create, modify http responses and requests) with incoming or outgoing data. You can do this by graphic interface.


JUNITs + maven build + Spring MVC REST
This is not out of the box app for testing. It's 3 separate products which can be used together to help you create test application for REST API.

JUNIT is a simple framework to write repeatable tests for the Java programming language. It is an instance of the xUnit architecture for unit testing frameworks. Alternative to JUnit can be TestNG.

Maven build refers to Apache Maven. Maven can manage a project's build, reporting and documentation from a central piece of information.

Spring MVC REST refers to REST support in Spring's MVC framework. Spring is modular application development framework for enterprise Java. It allows you to focus on your business problem rather than the plumbing code that connects components and systems. Spring MVC REST allows you to annotate java methods (that contains just business logic) which are supposed to communicate using REST. The framework then generates plumbing code for these methods and add it to you application.
Alternative to Spring MVC REST can be Jersey.

By using these three tools you gain the big advantage of writing your testing app completely suited for your needs. That's of course the biggest disadvantage too as you get absolutely no functionality out of the box.

Summary
So that was some apps for testing. Except JUNITs + maven build + Spring MVC REST which is too general, they all have one thing in common. They just inspect and evaluate HTTP requests/responses and compare them against expected values you entered. They doesn't lead you to create truly RESTful API, nor they test if it is really RESTful.

We can ask ourselves if we can combine these apps to solve more then what single app offer. Well only particularly. SoapUI has great GUI for creating request and view responses. JMeter is shifted more to performance testing. FitNesse offers wiki based UI for inserting tests. Fiddler can control packet flows between your app and world. But even if we combine them, we still just get evaluation of response against expected response. We are still not able to test REST constraints.

So if we are going to create new testing app. Based on all these findings we can say we don't need another request/response inspector. We have to focus on testing REST constraints compliance and let other tools do the "expected response evaluation".

Is it possible to automatically test REST constraints?

In this post, I am going to look at testability of REST constraints. What interests me most is if we can test them one by one automatically? Are we able to write bot, program or some test framework which can tell "yes, your api is/isn't RESTful"? I am going to focus on REST constraints implemented on HTTP protocol.

You can take look at the constraints of REST and what benefits they brings at my previous post.

Constraints we can test automatically:

Cache constraint can be checked by looking at HTTP headers Cache-Control, Expires, Last-Modified. For fulfillment of this constraint, every response from server should contain them.
There is actually an online tool - redbot which analyzes the headers and tells about cacheability (among other things).


Constraints we maybe can test automatically:

Client-Server - HTTP itself is client-server. But we should somehow test if server just listens and wait for requests. My proposal: if we obtain a response after our client send request is it satisfying to tell this constraint is met?

Uniform Interface - my proposal: if this condition is met, the generic client should be able to go through all server's services (from one starting point) and used them without any special client's implementation. The client should only use standardized methods with communicating with server. 


Constraints we don't know how to test automatically:

Null style - we would have to test if some of "all other constraint" isn't present in the app. That's pretty huge number of tests ;-)

Stateless -  the programmer of the server can probably tell if it is stateless but if the server is black box for us, how to test if it keeps some state? Perhaps we can send same requests in a row and assume we always get same answer?

Layered System - the programmer of the server can probably tell if his system is layered, but how can we test it if server is black box for us?

Code-On-Demand - just optional constraint and again, the server programmer can tell if the server is capable of that.


Conclusion:

We can test one condition. We maybe can test another two. We cannot test another four but three of that four can be answered by programmer of server.

REST constraints and their impact on web application

What can be gained by making the api of our web application RESTful? What are the constraints of REST and what advantages can our web application loose when we decide not to follow some of REST constraints?
The answers to these questions lies in Roy's T. Fielding dissertation so I'll just list them here.

Null style: means there are no other constraints then following.

Client-Server: client wants something, server has it and can be asked for it by client. Still client and server can evolve independently.
Fulfillment of Client-Server constraint improves portability of the user interface across multiple platforms and improve scalability by simplifying the server components. And components can evolve independently.

Stateless: the request from client to server must contain all information so the server can understand the request. In other words, only client keeps information about it's state.
Fulfillment of stateless constraint gives our applications following three traits:

• Visibility - monitoring system does not have to look beyond a single request in order to determine the full nature of the request.
• Reliability - eases the task of recovering from partial failures.
• Scalability - server can quickly free resources and doesn't have to manage resource usage across requests.

Cache: all the data within response can be labeled as cacheable or non-cacheable.
Fulfillment of cache constraint is improving network efficiency, scalability and user perceived performance by reducing the average latency of interactions.

Uniform Interface: implementations are decoupled from the services they provide. Informations are transfered in standardized form. To obtain an uniform interface we need to define interface constrains. For REST they are:

• identification of resources: we can address desired information and we can recognize them.
• Manipulation of resources through representations: we are manipulating with representation of resource, not with resource itself and in a format which suits the client. Although it can be same as original resource.
• Selfdescriptive messages: the messages consist of data, metadata describing the data, and optionally metadata describing metadata.
• Hypermedia as the engine of application state: server is master of it's resources. The resource implementation details are hidden behind the interface. Server publish it's interfaces as resource hiearchy. Server does not tell which method is supposed to use with which URI.

Fulfillment of Uniform Interface constraint simplifies system architecture and improving visibility of interactions. It ensures the independent evolvalibility because implementations are decoupled from services they provide

Layered System: whole system is divided to layers. The component of each layer can "see" only it's immediate layer.
Fulfillment of Layered System constraint improves internet-scale requirements. Bounding overall system complexity and promoting substrate independence.

Code-On-Demand (optional constraint): client functionality can be extended by downloading and executing code (scripts for example).
Needed features doesn't have to be implemented in client at the time of compilation.