Handling Groovy scripts in Mule ESB

In my work project we had to make plenty of XML manipulations on some complex structures. As Mule supports Groovy (scripts) natively, we decided to make use of Groovy's XMLSlurper.
Two things I wanted to have covered, before I begin: How to easily handle those scripts in Mule Studio and how to unit test them.

1. Groovy scripts in Mule Studio

Mule, when it comes to Groovy components, can embed script code inside the Mule configuration file or use an external script file. Of course, at all times, I don't want to clutter my Mule code with embedded scripts. The only exception I make, is when I need to do something simple and I'm unable to use Mule Expression Language to achieve that (i.e. throwing an exception directly from flow - JIRA).

First thing I did was installation of Groovy plugin for the IDE. Latest Mule Studio version is 3.5.0, which is based on Eclipse 3.8. Update site URL for the plugin is: http://dist.springsource.org/release/GRECLIPSE/e4.2/. My example project was built on Mule ESB 3.4.0. It uses Groovy 1.8.6, hence we have to add an extra Groovy compiler when installing plugin and we have to switch it on in Groovy preferences afterwards.

Next, we need to think about where to locate our scripts. There are several possibilities, but I find most convinient to put them in src/main/scripts in a subfolder named as a Mule configuration file, which uses it. Then, we need to force IDE to be as useful as possible, especially the m2e plugin built in Mule Studio. I always work on Maven-supported Mule projects and strongly recommend to do the same.

• Enable Groovy scripts folders in Groovy preferences and add src/main/scripts and src/test/scripts patterns (note that Compiler version is already set to 1.8.6):

• Now we need to make sure that Maven will put those scripts in proper location. Ideal for me, is to put them in classes/scripts directory. It will cause nice separation with other project resources. Achieving that appeared to be simple. In pom.xml specify resources in the <build> part:

<build>
  <resources>
    <resource>
      <directory>src/main/resources</directory>
    </resource>
    <resource>
      <directory>src/main/scripts</directory>
      <targetPath>scripts</targetPath>
    </resource>
  </resources>
  ...
</build>

Maven's build <resources/> element is described here. When doing research about the above I found out something interesting. With the default structure of pom.xml file generated by Mule Studio, all Mule configuration files are duplicated in the built zip file. Sample Mule application structure generated by Maven would look like:

/
 \- classes
  \- mule-config.xml
  |- mule-deploy.properties
  |- mule-app.properties
 |- lib
 |- mule-config.xml
 |- mule-deploy.properties
 |- mule-app.properties

All these bolded files are redundant (per Mule's application format reference). Responsible for that mess is Build Helper Maven Plugin:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>1.7</version>
  <executions>
    <execution>
      <id>add-resource</id>
      <phase>generate-resources</phase>
      <goals>
        <goal>add-resource</goal>
      </goals>
      <configuration>
        <resources>
          <resource>
            <directory>src/main/app/</directory>
          </resource>
        </resources>
      </configuration>
    </execution>
  </executions>
</plugin>

If I remove that plugin part, everything still seems to be working perfectly fine and my project's output structure is neat and clean:

/
 \- classes
  \- scripts
   \- GroovyScriptTesting
    \- CalculateSquareNumber.groovy
  |- log4j.properties
 |- GroovyScriptTesting.xml
 |- mule-deploy.properties
 |- mule-app.properties

It looks like maven-mule-plugin is doing all the necessary work for src/main/app folder. So, if you know why the build helper plugin is set up in the pom.xml in the first place or if you're having any problem without it, please let me know.

In the end, with everything in place, my project in Mule Studio should look like this:

and we are ready to go to the second part.

2. Unit testing Groovy scripts

Mule provide direct access to Mule context objects and variables in Groovy scripts the same way as it does for MEL. In terms of testing, it's important to decide and be consequent about how we are accessing Mule variables:

If I'm sure that the variable will be defined during script execution, I access them directly (as 'number' in the example):

def square = number.toInteger() * number.toInteger()

If it's possible that the variable won't be set up (can be a part of the script logic - to check for variable presence), then I would try to get it from the message context (it's always available):

def number = message.getInvocationProperty('number')

and then do some null checks etc. Otherwise, I would get Groovy's MissingPropertyException.
Setting variables via scripts always needs to happen using message.setProperty method.

Let's look at this simple Groovy script (CalculateSquareNumber.groovy):

def square = number.toInteger() * number.toInteger()
message.setInvocationProperty('squareNumber', square)

and it's unit test (CalculateSquareNumberTest.java):

@RunWith(JUnitParamsRunner.class)
public class CalculateSquareNumberTest {

	private static final String PAYLOAD = "payload";

	@Test
	@Parameters(method = "numbersAndSquares")
	public void test(int number, int square) throws Exception {
		
		Binding binding = new Binding();
		binding.setVariable("number", number);
		binding.setVariable("message", TestMuleMessage.withPayload(PAYLOAD));
		
		GroovyShell shell = new GroovyShell(binding);
		shell.evaluate(getFile("/scripts/GroovyScriptTesting/CalculateSquareNumber.groovy"));
		MuleMessage message = (MuleMessage) binding.getVariable("message");
		
		assertThat((Integer)message.getInvocationProperty("squareNumber"), is(square));
	}
	
	@SuppressWarnings("unused")
	private Object[] numbersAndSquares() {
		return $(
             $(3, 9),
             $(5, 25),
             $(10, 100),
             $(12, 144)
        );
	}
	
	private File getFile(String pathToFile) throws URISyntaxException, FileNotFoundException {
		URL url = CalculateSquareNumberTest.class.getResource(pathToFile);
		if (url == null) {
			throw new FileNotFoundException("Couldn't find: " + pathToFile);
		}
		return new File(url.toURI());
	}
}

To evaluate Groovy scripts in our test we need to use GroovyShell along with Binding's setVariable() method to make variables available. As you can see it's pretty easy, the trickier part is how to pass proper MuleMessage instance to our test. To do that, I have prepared helper TestMuleMessage class, which is preparing message with default Mule context:

public class TestMuleMessage {

	public static MuleMessage withPayload(Object payload) {
		
		MuleContextFactory contextFactory = new DefaultMuleContextFactory();
		MuleContext muleContext = null;
		try {
			muleContext = contextFactory.createMuleContext();
		} catch (InitialisationException e) {
			e.printStackTrace();
		} catch (ConfigurationException e) {
			e.printStackTrace();
		}
		
		return new DefaultMuleMessage(payload, muleContext);
	}
}

Another useful thing in the test is the helper getFile() method which retrieves script file not from hardcoded full path, but from classpath instead, which is always a better idea. Usually, you would put it in some test util class.

Your main target for assertions is MuleMessage object, which can be retrieved using Binding getVariable() method. There you can check for variables' values or message payload itself.

As you can see, keeping few simple rules, can make work with Groovy scripts (in Mule) a pleasant, easily testable experience. Sample project, I prepared, is using Quartz endpoint and Groovy script to count and display square number of number 5 :) It is available here.

Exposing RESTful interface with Mule pt.2

In previous part of this article I've presented two ways of building REST API on Mule ESB: using Mule REST Module and handling HTTP properties manually.

This time I want to present achieving the same result using Mule Jersey Module with addition of
Component Bindings - feature of Mule, which enables us to leave Java and go back to Mule flow processing.

Presented example was tested against Mule ESB 3.4.0 EE.

3. Jersey REST

Flow overview:

Unfortunately, not much to see in this graphical overview - devil is in the details :)

Jersey REST Component is the official recommendation for exposing REST Services on Mule ESB. Usually, this reference implementation of JAX-RS would be everything we need to achieve the goal. However, in terms of Mule, it ties us heavily to Java code, which is not what we are especially looking for. To overcome this we can use Component Bindings.

Let's go back to our example. Staring with the simple Jersey service class:

@Path(value = "/")
public class JerseyRestService {

 private FlowProcessing flowProcessing;

 @GET
 @Path(value = "/client/{accountID}/{userID}/get")
 public Response processGET(@PathParam("accountID") String accountId, @PathParam("userID") String userId) {
  String result = flowProcessing.processGET(accountId, userId);
  return Response.ok(result).build();
 }

 @POST
 @Path(value = "/client/{accountID}/{userID}/get")
 public Response processPOST(@PathParam("accountID") String accountId, @PathParam("userID") String userId, String body) {
  String result = flowProcessing.processPOST(accountId, userId, body);
  return Response.ok(result).build();
 }

 public void setFlowProcessing(FlowProcessing flowProcessing) {
  this.flowProcessing = flowProcessing;
 }
}

Nothing fancy here. JAX-RS plain and simple. Interesting part is the FlowProcessing interface. It looks like it's wrapping whole business logic processing. By using Component Bindings we can make any Mule's outbound endpoint behave as an implementation of that interface. Hence, we can make use of VM endpoints and move the processing from Java back to Mule flows:

<jersey:resources doc:name="REST">
  <component class="pl.poznachowski.jerseyrest.JerseyRestService">
    <binding interface="pl.poznachowski.jerseyrest.FlowProcessing" method="processGET">
      <vm:outbound-endpoint exchange-pattern="request-response" path="vmProcessGET" />
    </binding>
    <binding interface="pl.poznachowski.jerseyrest.FlowProcessing" method="processPOST">
      <vm:outbound-endpoint exchange-pattern="request-response" path="vmProcessPOST" />
    </binding>
  </component>
  <jersey:exception-mapper class="pl.poznachowski.jerseyrest.BadURIExceptionMapper" />
</jersey:resources>

As seen above, everything we need to make it work is to provide binding element specifying the interface, interface's method and outbound endpoint, which should be called. Few things to remember:

• Don't forget to write setter for the interface in the Jersey class. 
• Make sure that request and response of the interface and endpoint matches 
• It is possible to have the method return MuleMessage. It lets Java component have access to whole message, not only payload.

Flows with the "business logic" for POST and GET methods processing are described below:

<flow name="ProcessGetFlow" doc:name="ProcessGetFlow">
  <vm:inbound-endpoint exchange-pattern="request-response" path="vmProcessGET" doc:name="VM" />
  <transformer ref="populateVariables" doc:name="Populate variables"/>
  <set-payload value="Processing GET with account id: #[accountID] and user id: #[userID]" doc:name="Set Payload" />
</flow>
<flow name="ProcessPostFlow" doc:name="ProcessPostFlow">
  <vm:inbound-endpoint exchange-pattern="request-response" path="vmProcessPOST" doc:name="VM" />
  <transformer ref="populateVariables" doc:name="Populate variables"/>
  <set-payload value="Processing POST with account id: #[accountID] and user id: #[userID] and body: #[payload]"
doc:name="Set Payload" />
</flow>

Input parameters of the FlowProcessing interface methods comes in the Mule flows as an Object array. We can map them into Mule parameters by writting and setting global transformer:
<custom-transformer class="pl.poznachowski.jerseyrest.PopulateVariables" name="populateVariables" doc:name="Java" />
and reusing it in the flows (<transformer ref="populateVariables" />).

PopulateVariables Java transformer:

public class PopulateVariables extends AbstractMessageTransformer {

 @Override
 public MuleMessage transformMessage(MuleMessage message, String outputEncoding) throws TransformerException {

  Object[] args = message.getPayload(Object[].class);
  message.setInvocationProperty("accountID", args[0]);
  message.setInvocationProperty("userID", args[1]);

  // For POST method
  if (args.length > 2) {
   message.setPayload(args[2]);
  }

  return message;
 }
}

We have everything in place to make it work. One thing I don't like is that Jersey for requests with URL not met is returning 404 HTTP status (Not Found). I find 400 status (Bad Request) more appropriate in such case and I'd like to keep 404 reserved for situations where URL was met, but resource was not found. Satysfying that requirement is fairly simple. We need to add a custom exception mapper in Jersey definition:
<jersey:exception-mapper class="pl.poznachowski.jerseyrest.BadURIExceptionMapper" />
with implementation:

public class BadURIExceptionMapper implements ExceptionMapper<NotFoundException> {

  public Response toResponse(NotFoundException exception){
      return Response.status(Response.Status.BAD_REQUEST).entity("Unknown resource: " + exception.getNotFoundUri().toString()).build();
  }
}

Note: It is possible to catch exceptions thrown in the binded outbound endpoints. To make that possible we just need to declare exception in binding interface method.

To test described solution I used the same set of tests as in previous post. However I was having problems testing successful scenarios using MUnit:

ERROR 2013-10-17 22:30:02,434 [main] org.mule.exception.DefaultMessagingExceptionStrategy:

********************************************************************************

Message               : Failed to invoke JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Component that caused exception is: JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Message payload is of type: String

Code                  : MULE_ERROR--2

--------------------------------------------------------------------------------

Exception stack is:

1. The required object/property "muleContext" is null (java.lang.IllegalArgumentException)

  org.mule.DefaultMuleMessage:292 (null)

2. Failed to invoke JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Component that caused exception is: JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Message payload is of type: String (org.mule.component.ComponentException)

  org.mule.component.AbstractComponent:148 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/component/ComponentException.html)

It seems that muleContext is not propagated correctly with MUnit and reflective proxy classes. Switching test class to use 'official' FunctionalTestCase instead of MUnit worked like a charm.

Full example at GitHub: JerseyREST

That were all the solutions for exposing REST Services I can think of.
In my work project, I'm using the REST Module and it's doing its job well. Thus, if you want build REST API and use all of the nifty Mule features  I would suggest going with the Router Module.
If you prefer to be more Java-centric, make use of Mule for handling integration matter only or to use Mule flows to a smaller extent then Jersey approach will fit in.

For more information about Component Bindings I encourage you to read Mule's well-written blog post explaining the feature.

Exposing RESTful interface with Mule pt.1

Update (20.10.2013) - When not supported HTTP method is used, return more appropriate HTTP status (405 instead of 400).

As it's my first blog entry I'd like to welcome everyone. If you'd like to know more about me click on the tab above. If not.. I'll go straight to the point.

Recently, I had to create Mule's application, which exposes itself via a simple RESTful API. However, when it comes to REST, Mule ESB seems to be quite limited in viable options.
The only, official approach is to use the REST Component, which relies on Jersey, which is the Reference Implementation of JAX-RS. Sounds good, but it ties you to the Java code instead of having fun with the Mule's message processing :) This can be overcome as well, but still some Java needs to be written.

Fortunately, there are some other possibilites, which would make simple REST API creation easier. Nevertheless, I'd like to present you all the options (of which I'm aware of) with description and a working example.

All presented examples were tested against Mule ESB 3.4.0 EE. However, they should run without any problems on CE and even on previous versions.

1. "Poor man's" REST:

First approach is the most straightforward one - handling HTTP properties available in Mule manually. Quick overview over HTTP properties is available here - (1.4.7 Use Case) in archived Mule Developer Resources.

In that case, we need to take care of detecting the HTTP method, URL and fetching URL variable parameters by ourselves. We can accomplish that using Mule's choice component and setting flow variables (if needed).
An example will accept URL (on 8088 port) as follows: /client/{accountID}/{userID}/get on both GET and POST HTTP methods, where {accountID} and {userID} are the variable parameters.

General overview of the flow:

First step is not necessary, but helpful for quick suppresion of favicon.ico requests when accesing service via browsers:

<message-filter doc:name="Filter favicon">
  <not-filter>
    <wildcard-filter casesensitive="true" pattern="/favicon.ico">
  </wildcard-filter></not-filter>
</message-filter>

To match URL let's use choice component with regular expression:

<choice doc:name="Choice">
  <when expression="#[regex('/client/\\w+/\\w+/get/?', message.inboundProperties['http.request.path'])]">
   ...
 </when>
  <otherwise>
    <http:response-builder status="400" doc:name="Return 400 For bad URL">
      <set-payload value="Unknown resource: #[message.inboundProperties['http.request.path']]" />
    </http:response-builder>
  </otherwise>
</choice>

We retrieve URL path using Mule's inbound property - 'http.request.path', then match it with the regexp '/client/\\w+/\\w+/get/?'. If we don't have a match we just simply return 400 HTTP status.
Otherwise, we can proceed and determine which HTTP method was used to send the request:

<choice doc:name="Choice">
  <when expression="message.inboundProperties['http.method'] == 'GET'">
    <flow-ref name="RetrievingParameters" doc:name="Retrieve Parameters"/>
    <flow-ref name="ProcessGET" doc:name="Process GET"/>
  </when>
  <when expression="message.inboundProperties['http.method'] == 'POST'">
    <flow-ref name="RetrievingParameters" doc:name="Retrieve Parameters"/>
    <flow-ref name="ProcessPOST" doc:name="Process POST"/>
  </when>
  <otherwise>
    <http:response-builder status="405" doc:name="Return 405 For bad HTTP method">
      <set-payload value="Unknown HTTP method: #[message.inboundProperties['http.method']]" />
    </http:response-builder>
  </otherwise>
</choice>

This time we are checking another inbound parameter - 'http.method'. If it's not the method we want to proceed with, we return 405 HTTP status (Method Not Allowed).

For all supported methods we want to fetch the URL parameters. Hence, this functionality is extracted to a reusable sub-flow:

<sub-flow name="RetrievingParameters" doc:name="RetrievingParameters">
  <set-variable variableName="accountID" value="#[StringUtils.splitAndTrim(message.inboundProperties['http.request.path'], '/')[1]]" doc:name="Set Account ID"/>
  <set-variable variableName="userID" value="#[StringUtils.splitAndTrim(message.inboundProperties['http.request.path'], '/')[2]]" doc:name="Set User ID"/>
</sub-flow>

As I'm not good at regular expressions, I decided to go with Mule's StringUtils method, which splits the URL and gets me the parameters I'm looking for.

To avoid providing fully qualified name of the class over and over again, we can define global import for Mule expressions:

<configuration doc:name="Configuration">
  <expression-language>
    <import class="org.mule.util.StringUtils" />
  </expression-language>
</configuration>

That is basically everything we needed. We can now process GET / POST requests. For readability and maintainability reasons we will do that in separate flows:

<flow name="ProcessPOST" doc:name="ProcessPOST">
  <http:response-builder status="200"doc:name="Return OK">
    <set-payload value="Processing POST with account id: #[accountID] and user id: #[userID] and body: #[payload]" doc:name="Set Payload"/>
  </http:response-builder>
</flow>
<flow name="ProcessGET" doc:name="ProcessGET">
  <http:response-builder status="200"doc:name="Return OK">
    <set-payload value="Processing GET with account id: #[accountID] and user id: #[userID]" doc:name="Set Payload"/>
  </http:response-builder>
</flow>

 
To test my service I've prepared a test class with couple of Mule functional tests. Instead of using default FunctionalTestCases, I suggest using MUnit. From what I know, it is supposed to replace current Mule testing solution in future. With MUnit you can write your tests using Java (JUnit) or XML (Mule code). It has features like mocking endpoints, processors and ability to call flow directly, skipping inbound endpoints. However, development of MUnit seemed to slow down recently.
To keep the test concise, I used JUnitParams extension, as I wanted to use same test methods, but with different input (URLs).

Additionally, we could've checked for content-type etc., but I didn't want to clutter the code with insignificant details.
Having in mind next example, I'm not exactly sure, if anyone would opt for this approach. But still, there it is, probably for demonstration purposes only ;)

Full example at GitHub: PoorMansREST

2. REST Router

To make our lives easier Mule REST Router Module has been developed. Code is available at GitHub.
Almost everything we did in previous approach is wrapped into one message processor. Installation and configuration is well explained in the links provided.

Flow overview:

Basically, to implement same behaviour, we need to configure rest-router as follows:

<rest-router:router templateUri="/client/{accountID}/{userID}/get">
  <rest-router:get>
    <flow-ref name="ProcessGET" doc:name="Process GET" />
  </rest-router:get>
  <rest-router:post>
    <flow-ref name="ProcessPOST" doc:name="Process POST" />
  </rest-router:post>
</rest-router:router>

REST Router automatically assign parameters specified in templateUri to flow variables. However, I found out that it passes through empty parameters, so it is advisable to validate them explicitly in your code.

Outstanding part is to cover invalid URLs and return something meaningful. For requests not matching our template we return standard 400 HTTP status (placed just after </rest-router:router>) :

<http:response-builder status="400" doc:name="Return 400 For bad URL">
  <set-payload value="Unknown resource: #[message.inboundProperties['http.request.path']]" />
</http:response-builder>

For HTTP methods not supported REST Router should (per documentation) throw UnsupportedHttpVerbException. However, there is an issue I submitted here. Unfortunately, the project looks abandoned as nobody cared :) I did some more research and it appears that the issue is not in the REST Router itself, but how the DevKit java code is generated.
There are two solutions for that:
1) Always implement whole set of methods in REST Router and return appriopriate message / status / throw exception in those you don't want to support.
2) As I don't like to have redundancy in my code I used a dirty hack of replacing compiled class with a fixed one directly in the jar file. Amended version of the module is available here. Just install it in your local Maven repository and switch module version in your pom.xml file to 1.2-fixed.

Fix description:

HTTP method recognition is implemented as in the following snippet: (RestRouterModule.java)

if (get != null && method.equalsIgnoreCase("get")) {
 return get.processWithExtraProperties(properties);
} else if (put != null && method.equalsIgnoreCase("put")) {
 return put.processWithExtraProperties(properties);
} else if (post != null && method.equalsIgnoreCase("post")) {
 return post.processWithExtraProperties(properties);
} else if (delete != null && method.equalsIgnoreCase("delete")) {
 return delete.processWithExtraProperties(properties);
} else if (patch != null && method.equalsIgnoreCase("patch")) {
 return patch.processWithExtraProperties(properties);
} else {
 throw new UnsupportedHttpVerbException(method);
}

Variables: get, put, post... are instances of NestedProcessor class, marked @Optional.
However, code generated by DevKit always propagates those variables to the RestRouterModule as instances of NestedProcessorChain (RouterMessageProcessor.java):

final NestedProcessor _transformedGet = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) get));
final NestedProcessor _transformedPut = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) put));
final NestedProcessor _transformedPost = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) post));
final NestedProcessor _transformedDelete = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) delete));
final NestedProcessor _transformedPatch = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) patch));

Those _transformed method variables are passed into the RestRouterModule class, hence it always tries to process the processWithExtraProperties(properties) method and eventually finishes with NullPointerException as the last argument in NestedProcessorChain contructor is null.

My fixed version of RouterMessageProcessor.java includes following change:

final NestedProcessor _transformedGet = (get == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) get));
final NestedProcessor _transformedPut = (put == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) put));
final NestedProcessor _transformedPost = (post == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) post));
final NestedProcessor _transformedDelete = (delete == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) delete));
final NestedProcessor _transformedPatch = (patch == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) patch));

Not much to explain here. Now, if a REST module HTTP method is not defined it will propagate null to RestRouterModule.java and throw UnsupportedHttpVerbException as expected.

In the flow we use standard choice exception strategy and catch the UnsupportedHttpVerbException to return 405 HTTP status:

For testing I used the same set of tests as in previous example. As you can see, it's much quicker and cleaner approach that saves you a lot of time and lines of code :)

Full example (using modified version of the module) at GitHub: RestWithRouter

In the next part I will present how to achieve the same, but using Jersey with writing as little Java as possible. Go here for the second part!