Saturday, February 14, 2015

How to make Jenkins speak Git Flow?

It's been a while and quite a lot has changed since then. I moved to a different company, which means I won't be playing around with Mule for now. On the other hand, I have several new things to play with.
My current company lacked of proper versioning capability of their products. Hence, my first task was to help with that matter. The plan was to be able to utilize Jenkins for modules releases (via Maven) with keeping all powers of a Git Flow branching convention. What's more, I extended existing functionality of building Debian packages and used Nexus as a native .deb repository.

Please go check the Neoteric's blog for the full story.

Thursday, April 3, 2014

MUnit testing, Mule best practices and more...

Quite long time passed, since my last update. I hope that this post will successfully fill the resulting gap and you will find reading it worthwhile.

TOC:

1. A little bit too long of a foreword...

When I first met Mule there was that thing, that made me feel a little bit discouraged. There was only residual information about best practices on how to develop Mule applications and even less advice on how to effectively test them. All Mule examples provided here and there on the Internet are presented in a deploy-click-check if works manner. Next thing is that all demos are mostly simple, one-flow applications. Question about how to cope with larger, complex configurations was left unanswered.
I'll try to shed some light on those subjects. For starters, there is some great writing by Ryan Hoegg available at Confluex blog. I'll try to elaborate more on some topics brought up by Ryan, but focusing mainly on testing part using MUnit framework.
When talking about testing in Mule environment it is worth to emphasize that two types of tests exist here. Following the Mule documentation naming convention, first type are unit tests, which cover your self-written message processors (transformers, components etc.) Mule's current testing capabilities are more than enough to handle those cases pretty well. In terms of testing Groovy script components here is something I wrote previously.
The other are functional tests - our main point of interest for this blog post.
I'd like to call them flow tests. Scope of these kinds of tests are pretty wide. How you approach them is up to you. You can test any given part of your application (depending on how you designed and structured your flows):

  • From testing integration between two adjacent message processors
  • via business logic branches (capable of having mocked out any part of the process - external service calls, connections to DB, sending MQ messages etc.)
  • to full system testing with all external services calls on board.

Obviously, the best is to have mix of those, excluding the latter from your automated build test suite. We don't want to call actual, external services each build, do we? Here is an idea of how you can separate them.

For all that MUnit comes in handy...

2. MUnit introduction

MUnit is a next generation Mule testing framework. It is still under development, but its current state (3.4-M5) is more than enough to persuade me to use it in my work projects. In fact, I'm using it since the first day I discovered it, as the current testing framework lacks flexibility in my opinion.
Plenty of MUnit features are not easily available to the current, official version (i.e. mocking flows and outbound endpoints is cumbersome and requires a lot of effort). What's more, MUnit comes in two flavours:

  • Well-known Java approach, where you write your tests as you would normally do (via JUnit).
  • Mule XML code with MUnit schema.

One thing needs mentioning. This blog post is strictly Java-centric, I won't tell you anything about the XML way of writing MUnit tests. If that was not what you were looking for, I'm sorry to disappoint you.
The reason is simple - I'm not a fan of XML and the Java approach suits me much better. Nevertheless, everything MUnit can do with XML is possible using Java (and the other way round). That's, at least, what specification states :) (I haven't checked that).

MUnit is getting more recognizable lately (eventually it ought to replace current testing framework). Latest Mule (3.5 Early Access version) starts integrating MUnit with Studio heavily. More information about MUnit capabilities is available at project's wiki page, but I'll try to cover most of its powers here.

3. Think about your flow design

I'll try to outline some guidelines I now follow. They emerged from my  ~year Mule integration experience. Note: I can't guarantee they will solve all your problems nor they will fit each of possible use cases. I'm just saying that they helped me survive some of the bad times I had. What's even more important, I'm not even sure if they are the best I can get from my cooperation with Mule and Studio. If you have your own practices, questions, suggestions I'd like to hear them!

In the beginning I want to emphasize that Mule's code it's even easier to get messy with than with Java (at least for me). Hence, having good test coverage is crucial. How you design your flows implies the ease of testing them. What's more, I really recommend to start your work flow in a TDD manner and let you flow design be guided by tests. From my experience, I can say, it pays off greatly later on.

Keep your flows short, make sure that each flow follows single responsibility principle. For complex solution split your configuration file (.xmls containing flows) into smaller ones. Think in functionality and business logic terms. I'm used to having one configuration file with main flow (with source component) controlling the whole process and others implementing bigger chunks of business logic (i.e. sending attachments with gathered information, fulfilling customer data) Of course each of these configuration files can contain many flows / subflows. Mule by mule-deploy.properties file (config.resources attribute) knows, which files comprise whole application set-up.

Let's say we have a requirement to expose a service via HTTP endpoint, which needs to perform some complex validation logic (on HTTP Headers, Request Body etc.), then transform the provided data into a proprietary structure, backing it up with data coming from external REST Service. Eventually send a notification email to anyone who it may concern.
Example configuration can look like this:
a) main-flow.xml:

This main flow can accommodate whole application process into few single flow references, which can be easily read and understood. Every of the bigger, complex tasks was extracted to a different configuration file. With such structure we can execute end-to-end tests or test only main processing paths of full solution (when flow has choice components as presented) by mocking out all referenced implementations. To not litter the main flow, we extracted variables initialization part to a subflow. Those variables can be then used in processing data or sending email part.

b) validate-request.xml, process-customer-data.xml, send-email-notification.xml:


Each of that configuration files is responsible for different phase of the solution. We can do the same with them if the flows would get bigger: split into more compact flows / subflows or even into another configuration files if necessary.

At some point in time, you may come up with the idea, that part of application should be extracted as separate Mule application. Keeping your flows modularized, makes this task relatively easy.
What's also important (and it's our subject) is that it allows us to test parts of the application in isolation. With MUnit you don't need to have an inbound endpoint to call your flow. In fact, in MUnit you can access any flow directly (it disables all inbounds by default). You can test validation part by running your test against ValidateHTTPRequest flow or check if all variables were initialized properly by calling InitializeVariables subflow from within MUnit test.

Lastly, a word about flow 'packaging'. Unfortunately Studio can't accept the fact that I'd like to have my config files structured in subdirectories. The only appropriate place for them, according to Studio, is src/main/app. Whenever you try to put a flow config into a subfolder, Studio recreates it in 'root' directory..

4. Test constants

You will be using variables in your tests. A lot! Flow names, variable names, configuration files to name a few. Keeping those names as interface constants gives you two advantages:
1. When renaming any variable, you don't have to go through all tests and correct one by one - just rename once in the interface.
2. When you enable your interface class for static imports in Mule Studio, you can type those values in no-time using code completion assistance:


5. Assertions

To make sure that flows behave in the way we expect them to, there are two objects we are most interested in checking: MuleEvent and MuleMessage. These two are main targets for assertion expressions. MuleEvent is a basic entity which MUnit test returns as a result. it contains MuleMessage, attachments and session data.

I'm a big fan of fluent assertions, hence I strongly recommend giving them a try. AssertJ is the framework to look at. It's a successor of Fest Assert. Constantly under development, has plenty of contributors with a thriving community. Most valuable benefit of using AssertJ is the great number of assertion types and direct support from your IDE (code completion).

To ease work with Mule stuff, I've created small extension for AssertJ. It is available at my GitHub, here. Right now it's an early version, bound together with something I call property enricher (I'll cover that later on). It covers presence of Mule Message properties (of different scopes), payload checks etc. Not everything's included, but most common cases I encountered in my projects.
It's not available in any public repository (yet?). You can build it yourself or get jar and sources from here, then install them into your local maven repository manually (or preferably into your artefact repository). Then you follow these instructions, common for every other AssertJ extension:
1. Add the library to your project
2. Enable pl.poznachowski.mule.message.assertions.Assertions for static imports
3. Start using:
    a)
    
    b)

I think there is no need to go into details about available assertions. You can explore them on your own. Just one note here: When making assertions on MuleEvent object, make sure you 'traverse down' to asserting MuleMessage using hasMuleMessageWhich() method as the last check for MuleEvent, as you won't be able to go back to it in the same assertion statement.

6. MUnit test structure

MUnit test does not differ much than the previous functional Mule test. Simplest example is presented below:
package pl.poznachowski.munitblogsamples;

import static pl.poznachowski.mule.message.assertions.Assertions.assertThat;
import static pl.poznachowski.munitblogsamples.TestConstants.VALIDATE_REQUEST_FLOW;

import org.junit.Test;
import org.mule.api.MuleEvent;
import org.mule.munit.runner.functional.FunctionalMunitSuite;

public class FirstMunitTest extends FunctionalMunitSuite {

 @Test
 public void simplestTest() throws Exception {
  MuleEvent result = runFlow(VALIDATE_REQUEST_FLOW, testEvent("payload"));
  
  assertThat(result)
   .hasMuleMessageWhich()
     .hasStringPayload("payload");
 }
}

As you can see, it's an ordinary JUnit test extending FunctionalMunitSuite class.
There are two thing we need to do in our test:

  1. Prepare MuleEvent object as an input to our flow. We can do that by using provided testEvent(Object payload) method.
  2. Execute runFlow(String flowName, MuleEvent event) method specifying flow name to test against and event we just created in the first step.

Of course, to make our test sensible, we need to assert whether the expected result is equal to what the flow actually produced as an output.

MUnit, by default, scans your mule-deploy.properties file and load all configuration files it can find there to perform the test. It's rarely the case we'd like to have (except end-to-end testing). Main reason for avoiding that, is speed. We our tests to perform as fast as possible and there is no reason to include whole configuration when we are testing (i.e.) validation logic only. To accomplish that we need to override getConfigResources() method and provide configuration files, which should participate in a test:
public class FirstMunitTest extends FunctionalMunitSuite {

 @Override
 protected String getConfigResources() {
  return Joiner.on(',').join(MAIN_FLOW_CFG, VALIDATE_REQUEST_CFG);
 }

This method accepts String value with comma separated config file names. To produce such String I use Guava's Joiner class. All configuration files names should, of course :), be set up as interface constants (i.e. MAIN_FLOW_CFG = "main-flow.xml")

Another thing we need to know about MUnit is, that all inbound endpoints are skipped by default, meaning the process begin just right after when it would be intercepted by the source endpoint. However, it is still possible to use an inbound entry point for accessing flow in a test. There are two ways to achieve that:
If you enabled your inbound endpoints, then you can access your flows in the old way. Only thing you need to do is to obtain MuleClient from muleContext:
MuleClient client = muleContext.getClient();
and use the client instead of runFlow() method.

When working with custom connectors and mocking them out is not a desired behavior, there is a method you can use: haveToMockMuleConnectors(). In that case, we want to override it and return false.

In terms of packaging, I'm used to keeping my Mule tests in packages named after flow names (then subpackage them if necessary), i.e. pl.poznachowski.munit.example.mainflow

7. Property enricher

Properties are most widely used features of Mule. Hence, it won't be uncommon to test part of application with MuleMessage already containing some Mule properties or to mock a message processor to modify them (alter / create / remove).
When creating MuleEvent for testing purposes you only can set payload using provided testEvent() method. To make working with properties easier I've introduced something I call property enricher. You can fluently apply any kind of property on either MuleEvent object or MuleMessage:
 @Test
 public void simplestTest() throws Exception {
  
  MuleEvent testEvent = PropertyEnricher.enrich(testEvent("payload"))
       .withInvocationProperty(SOME_VARIABLE, true)
       .get();
  
  MuleEvent result = runFlow(VALIDATE_REQUEST_FLOW, testEvent);
  
  assertThat(result)
   .hasMuleMessageWhich()
   .hasStringPayload("payload")
   .hasInvocationProperty(SOME_VARIABLE).withValue(true);
 }

If you'd like to give it a try, it's a part of the same utility project I described before - available here.

8. Execution environments

I think it is always a good idea to parametrize any properties eligible to change between execution environments. This has been well explained in the Confluex blog post. Having prepared beforehand property files for each environment and then not to worry about it during deployment is something I really appreciate. But what if we want to make our flow behave differently between environments? It is also possible due to spring:import capability. Let me show you an example how we can roll it out.
Let's say that we are exposing a flow through a HTTP/HTTPS endpoint. For functional testing we want to use standard HTTP, but for pre-production (and production of course) we need to work with HTTPS.
To accomplish that, instead of having our endpoint 'hardcoded' in the flow, we'll use flow reference:

and set up spring import in the .xml config file:
 <spring:beans>
  <spring:import resource="classpath:my-endpoint-${connection.type}.xml" />
 </spring:beans>
You shouldn't care about Studio throwing errors. It is not a rare case, that Studio is complaining about something, which is perfectly ok for Mule ESB itself.
Don't forget to have {connection.type} parameter set up in wrapper.conf of each of the Mule instances. Assume to expect only two possibilities here: secured / unsecured. Then we need to prepare two separate configuration files (located on the classpath): my-endpoint-secured.xml and my-endpoint-unsecured.xml and provide flow reference implementations (secured with HTTPS component, unsecured with HTTP). Remember about having same flow name as in the flow reference. After that, we're done. Mule instance will pick appropriate setup, basing on the connection.type parameter provided.
One last thing we would need to remember about in terms of testing such constructs. MUnit need to know which HTTP type to test with. We can provide it by adding a system property in the getConfigResources() method:
 @Override
 protected String getConfigResources() {
  System.setProperty("connection.type", "unsecured");
  return "main-flow.xml";
 }
Of course those Strings should be extracted to constants, as you would use them a lot.

9. Mocking

It's time to talk about crucial element of flows testing. In the beginning we need to realize that every component in Mule is a message processor (endpoint, flow reference etc). That makes MUnit powerful tool, as it has the ability to mock any of such things. Lets decompose mock expression into two parts:

        1. Defining mock target

You always starts your mock expression with whenMessageProcessor() fluent method. Required parameter is the processor name (XML tag name in Mule configuration file).
Then, if our processor is within a namespace we need to apply .ofNamespace() method to our mock.
Lastly, if there is more than one element with same name (i.e. flow, sub-flow, endpoints) we need to add .withAttributes() method, which takes Map<String,Object> or varargs Attribute as an argument. The latter is easy to create with a static helper structure available: attribute(String name).withValue(Object value)

        2. Defining mock behaviour

When we're done with target declaration we need to specify how our mock should behave. There are a couple of options here:

.thenReturnSameEvent() - Means that message processor you mocked out will do nothing (outputs the same thing that came to the processor) during the flow processing.
.thenThrow(Throwable exception) - Mocked processor will throw specified exception.
.thenReturn(MuleMessage message) - This will return a MuleMessage object prepared beforehand. We can make use of available muleMessageWithPayload(Object payload) method to help ourselves here. Actually, I don't use thenReturn() method much, as it's not flexible enough. Especially when working with Mule properties and reusable flows.
.thenApply(MuleMessageTransformer transformer) - Most flexible method. Here we can prepare and use a Mule transformer, meaning we can do and return almost anything in our message (like quickly add properties using property enricher)

Example of flow mocking:
whenMessageProcessor("flow").withAttributes(attribute("name").withValue("ValidateHTTPRequest")).thenReturnSameEvent();
More examples can be seen here. (note the contains() part when mocking sub-flows)

While it seems as a great and powerful feature, let me tell you when and why not to use mocking. Referring to our example we have a MainFlow which has several flow references. I want to mock out all those references, to make a simple test of the choice component, whether it routes properly (to send email or not). My first thought with that approach in mind, was to provide only main-flow.xml to my MUnit test (I want to have rest of flows mocked). Unfortunately, that's not possible. Mule needs to set up all flows during start up, meaning it will fail saying that flow couldn't be found. To overcome that, we would still need to provide all flow constructs (despite mocking), which implies greatly increased Mule start up time. (referenced flows can be complex, compound of even more flows / subflows). Not to mention configuration hassle (did I included all required flows?).
Of course we want our tests to perform as quick as possible. Solution for that is the way I was mocking flows before MUnit era. You need to create simple testing mock flows (with same flow names as in production code) and provide them in getConfigResources() method instead of the originals (src/test/resources is a great place to store them), but be careful! When you work with test flows, don't use Mule Config Flow Editor. Otherwise, you would end up with duplicated tests flows in src/main/app folder. There's a JIRA ticket for that. It now says it has been resolved in 3.5 Dolomite version. Not sure about the code names... but in my current Mule Studio (Version: 3.5.0 Build Id: 201402101410) it is still not working properly.

10. Implementing mocked services

It may sometimes be useful to simulate some more complex behavior of an external service in a form of a Mule mocked service implementation. We can accomplish that by combining everything we covered so far.
First thing would be to make sure that service outbound address parameters are externalized to a properties file and that property file is made environment specific (via system property - refer to earlier mentioned Confluex blog post). Then we need to code actual mock implementation in a test flow (address parameters can be any). We also need to create property file with mock service address (as it would be an another environment configuration set up).

To wrap everything together we need to set system property (in getConfigResources()) to a value we are using in a property file indicating mocked service address. Remember also of adding this mocked service flow to a collection of flows, which shouldn't have their inbound endpoints disabled. Otherwise, your mocked service won't be accessible at all.

I know that MUnit also offers FTP, Mail and DB servers that you can spawn on the fly in your tests. However, I didn't try them out yet, so it's better you check MUnit wiki page for more information on that.

11. Verifying & spying

There two other MUnit features worth mentioning:
Veryfing is a way to assert whether a message processor has been called during test execution. Its semantic is very similar to mocking. Instead of whenMessageProcessor() method we need to use verifyCallOfMessageProcessor() providing processor name with its identification details (if necessary: ofNamespace(), withAttributes()) and then using available assertion methods: .atLeastOnce(), .atLeast(), .atMost(). Names are self-explanatory. With atLeast() and atMost() you need to provide integer value to make those assertions sensible.

Example validating that send email flow was called at least 2 times:
verifyCallOfMessageProcessor(FLOW).withAttributes(attribute(NAME).withValue("SendNotificationEmail")).atLeast(2);

Spying can be used to validate assumptions before any given message processor starts processing or just right after. Again you start the same as with verifying and mocking, but with spyMessageProcessor() method. When the target is acquired you can use .before() and .after() method passing any number of SpyProcess instances as an argument.
SpyProcess is an interface with one method signature: void spy(MuleEvent event). In that method implementation you can perform your assertions and check the payload in before or after phase.

Example checking variable presence and value, before and after using set-variable component:
spyMessageProcessor("set-variable").withAttributes(attribute("variableName").withValue("text"))
  .before(new SpyProcess() {
   @Override
   public void spy(MuleEvent beforeEvent) throws MuleException {
    assertThat(beforeEvent)
     .hasMuleMessageWhich()
      .hasInvocationProperty("text").notSet();
   }
  })
  .after(new SpyProcess() {
   @Override
   public void spy(MuleEvent afterEvent) throws MuleException {
    assertThat(afterEvent)
     .hasMuleMessageWhich()
      .hasInvocationProperty("text").withValue("hello");
   }
  });

12. Summary

That would be about everything I could come up with. If you have any questions, suggestions please let me know. Lastly, two 'pro' :) tips:
  • do not commit any .flow files (they are required for Studio only, and are being generated from corresponding .xml)
  • good log4j configuration can save you a lot of time and trouble!

Sunday, November 3, 2013

Handling Groovy scripts in Mule ESB

In my work project we had to make plenty of XML manipulations on some complex structures. As Mule supports Groovy (scripts) natively, we decided to make use of Groovy's XMLSlurper.
Two things I wanted to have covered, before I begin: How to easily handle those scripts in Mule Studio and how to unit test them.

1. Groovy scripts in Mule Studio

Mule, when it comes to Groovy components, can embed script code inside the Mule configuration file or use an external script file. Of course, at all times, I don't want to clutter my Mule code with embedded scripts. The only exception I make, is when I need to do something simple and I'm unable to use Mule Expression Language to achieve that (i.e. throwing an exception directly from flow - JIRA).

First thing I did was installation of Groovy plugin for the IDE. Latest Mule Studio version is 3.5.0, which is based on Eclipse 3.8. Update site URL for the plugin is: http://dist.springsource.org/release/GRECLIPSE/e4.2/. My example project was built on Mule ESB 3.4.0. It uses Groovy 1.8.6, hence we have to add an extra Groovy compiler when installing plugin and we have to switch it on in Groovy preferences afterwards.

Next, we need to think about where to locate our scripts. There are several possibilities, but I find most convinient to put them in src/main/scripts in a subfolder named as a Mule configuration file, which uses it. Then, we need to force IDE to be as useful as possible, especially the m2e plugin built in Mule Studio. I always work on Maven-supported Mule projects and strongly recommend to do the same.

  • Enable Groovy scripts folders in Groovy preferences and add src/main/scripts and src/test/scripts patterns (note that Compiler version is already set to 1.8.6):
  • Now we need to make sure that Maven will put those scripts in proper location. Ideal for me, is to put them in classes/scripts directory. It will cause nice separation with other project resources. Achieving that appeared to be simple. In pom.xml specify resources in the <build> part:
<build>
  <resources>
    <resource>
      <directory>src/main/resources</directory>
    </resource>
    <resource>
      <directory>src/main/scripts</directory>
      <targetPath>scripts</targetPath>
    </resource>
  </resources>
  ...
</build>
Maven's build <resources/> element is described here. When doing research about the above I found out something interesting. With the default structure of pom.xml file generated by Mule Studio, all Mule configuration files are duplicated in the built zip file. Sample Mule application structure generated by Maven would look like:
/
 \- classes
  \- mule-config.xml
  |- mule-deploy.properties
  |- mule-app.properties
 |- lib
 |- mule-config.xml
 |- mule-deploy.properties
 |- mule-app.properties
All these bolded files are redundant (per Mule's application format reference). Responsible for that mess is Build Helper Maven Plugin:
<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>1.7</version>
  <executions>
    <execution>
      <id>add-resource</id>
      <phase>generate-resources</phase>
      <goals>
        <goal>add-resource</goal>
      </goals>
      <configuration>
        <resources>
          <resource>
            <directory>src/main/app/</directory>
          </resource>
        </resources>
      </configuration>
    </execution>
  </executions>
</plugin>
If I remove that plugin part, everything still seems to be working perfectly fine and my project's output structure is neat and clean:
/
 \- classes
  \- scripts
   \- GroovyScriptTesting
    \- CalculateSquareNumber.groovy
  |- log4j.properties
 |- GroovyScriptTesting.xml
 |- mule-deploy.properties
 |- mule-app.properties
It looks like maven-mule-plugin is doing all the necessary work for src/main/app folder. So, if you know why the build helper plugin is set up in the pom.xml in the first place or if you're having any problem without it, please let me know.

In the end, with everything in place, my project in Mule Studio should look like this:


and we are ready to go to the second part.

2. Unit testing Groovy scripts

Mule provide direct access to Mule context objects and variables in Groovy scripts the same way as it does for MEL. In terms of testing, it's important to decide and be consequent about how we are accessing Mule variables:

If I'm sure that the variable will be defined during script execution, I access them directly (as 'number' in the example):
def square = number.toInteger() * number.toInteger()
If it's possible that the variable won't be set up (can be a part of the script logic - to check for variable presence), then I would try to get it from the message context (it's always available):
def number = message.getInvocationProperty('number')
and then do some null checks etc. Otherwise, I would get Groovy's MissingPropertyException.
Setting variables via scripts always needs to happen using message.setProperty method.

Let's look at this simple Groovy script (CalculateSquareNumber.groovy):
def square = number.toInteger() * number.toInteger()
message.setInvocationProperty('squareNumber', square)
and it's unit test (CalculateSquareNumberTest.java):
@RunWith(JUnitParamsRunner.class)
public class CalculateSquareNumberTest {

	private static final String PAYLOAD = "payload";

	@Test
	@Parameters(method = "numbersAndSquares")
	public void test(int number, int square) throws Exception {
		
		Binding binding = new Binding();
		binding.setVariable("number", number);
		binding.setVariable("message", TestMuleMessage.withPayload(PAYLOAD));
		
		GroovyShell shell = new GroovyShell(binding);
		shell.evaluate(getFile("/scripts/GroovyScriptTesting/CalculateSquareNumber.groovy"));
		MuleMessage message = (MuleMessage) binding.getVariable("message");
		
		assertThat((Integer)message.getInvocationProperty("squareNumber"), is(square));
	}
	
	@SuppressWarnings("unused")
	private Object[] numbersAndSquares() {
		return $(
             $(3, 9),
             $(5, 25),
             $(10, 100),
             $(12, 144)
        );
	}
	
	private File getFile(String pathToFile) throws URISyntaxException, FileNotFoundException {
		URL url = CalculateSquareNumberTest.class.getResource(pathToFile);
		if (url == null) {
			throw new FileNotFoundException("Couldn't find: " + pathToFile);
		}
		return new File(url.toURI());
	}
}

To evaluate Groovy scripts in our test we need to use GroovyShell along with Binding's setVariable() method to make variables available. As you can see it's pretty easy, the trickier part is how to pass proper MuleMessage instance to our test. To do that, I have prepared helper TestMuleMessage class, which is preparing message with default Mule context:
public class TestMuleMessage {

	public static MuleMessage withPayload(Object payload) {
		
		MuleContextFactory contextFactory = new DefaultMuleContextFactory();
		MuleContext muleContext = null;
		try {
			muleContext = contextFactory.createMuleContext();
		} catch (InitialisationException e) {
			e.printStackTrace();
		} catch (ConfigurationException e) {
			e.printStackTrace();
		}
		
		return new DefaultMuleMessage(payload, muleContext);
	}
}
Another useful thing in the test is the helper getFile() method which retrieves script file not from hardcoded full path, but from classpath instead, which is always a better idea. Usually, you would put it in some test util class.

Your main target for assertions is MuleMessage object, which can be retrieved using Binding getVariable() method. There you can check for variables' values or message payload itself.

As you can see, keeping few simple rules, can make work with Groovy scripts (in Mule) a pleasant, easily testable experience. Sample project, I prepared, is using Quartz endpoint and Groovy script to count and display square number of number 5 :) It is available here.

Sunday, October 20, 2013

Exposing RESTful interface with Mule pt.2

In previous part of this article I've presented two ways of building REST API on Mule ESB: using Mule REST Module and handling HTTP properties manually.

This time I want to present achieving the same result using Mule Jersey Module with addition of
Component Bindings - feature of Mule, which enables us to leave Java and go back to Mule flow processing.

Presented example was tested against Mule ESB 3.4.0 EE.

3. Jersey REST

Flow overview:

Unfortunately, not much to see in this graphical overview - devil is in the details :)

Jersey REST Component is the official recommendation for exposing REST Services on Mule ESB. Usually, this reference implementation of JAX-RS would be everything we need to achieve the goal. However, in terms of Mule, it ties us heavily to Java code, which is not what we are especially looking for. To overcome this we can use Component Bindings.

Let's go back to our example. Staring with the simple Jersey service class:
@Path(value = "/")
public class JerseyRestService {

 private FlowProcessing flowProcessing;

 @GET
 @Path(value = "/client/{accountID}/{userID}/get")
 public Response processGET(@PathParam("accountID") String accountId, @PathParam("userID") String userId) {
  String result = flowProcessing.processGET(accountId, userId);
  return Response.ok(result).build();
 }

 @POST
 @Path(value = "/client/{accountID}/{userID}/get")
 public Response processPOST(@PathParam("accountID") String accountId, @PathParam("userID") String userId, String body) {
  String result = flowProcessing.processPOST(accountId, userId, body);
  return Response.ok(result).build();
 }

 public void setFlowProcessing(FlowProcessing flowProcessing) {
  this.flowProcessing = flowProcessing;
 }
}

Nothing fancy here. JAX-RS plain and simple. Interesting part is the FlowProcessing interface. It looks like it's wrapping whole business logic processing. By using Component Bindings we can make any Mule's outbound endpoint behave as an implementation of that interface. Hence, we can make use of VM endpoints and move the processing from Java back to Mule flows:
<jersey:resources doc:name="REST">
  <component class="pl.poznachowski.jerseyrest.JerseyRestService">
    <binding interface="pl.poznachowski.jerseyrest.FlowProcessing" method="processGET">
      <vm:outbound-endpoint exchange-pattern="request-response" path="vmProcessGET" />
    </binding>
    <binding interface="pl.poznachowski.jerseyrest.FlowProcessing" method="processPOST">
      <vm:outbound-endpoint exchange-pattern="request-response" path="vmProcessPOST" />
    </binding>
  </component>
  <jersey:exception-mapper class="pl.poznachowski.jerseyrest.BadURIExceptionMapper" />
</jersey:resources>

As seen above, everything we need to make it work is to provide binding element specifying the interface, interface's method and outbound endpoint, which should be called. Few things to remember:

  • Don't forget to write setter for the interface in the Jersey class. 
  • Make sure that request and response of the interface and endpoint matches 
  • It is possible to have the method return MuleMessage. It lets Java component have access to whole message, not only payload.

Flows with the "business logic" for POST and GET methods processing are described below:
<flow name="ProcessGetFlow" doc:name="ProcessGetFlow">
  <vm:inbound-endpoint exchange-pattern="request-response" path="vmProcessGET" doc:name="VM" />
  <transformer ref="populateVariables" doc:name="Populate variables"/>
  <set-payload value="Processing GET with account id: #[accountID] and user id: #[userID]" doc:name="Set Payload" />
</flow>
<flow name="ProcessPostFlow" doc:name="ProcessPostFlow">
  <vm:inbound-endpoint exchange-pattern="request-response" path="vmProcessPOST" doc:name="VM" />
  <transformer ref="populateVariables" doc:name="Populate variables"/>
  <set-payload value="Processing POST with account id: #[accountID] and user id: #[userID] and body: #[payload]"
doc:name="Set Payload" />
</flow>
Input parameters of the FlowProcessing interface methods comes in the Mule flows as an Object array. We can map them into Mule parameters by writting and setting global transformer:
<custom-transformer class="pl.poznachowski.jerseyrest.PopulateVariables" name="populateVariables" doc:name="Java" />
and reusing it in the flows (<transformer ref="populateVariables" />).

PopulateVariables Java transformer:
public class PopulateVariables extends AbstractMessageTransformer {

 @Override
 public MuleMessage transformMessage(MuleMessage message, String outputEncoding) throws TransformerException {

  Object[] args = message.getPayload(Object[].class);
  message.setInvocationProperty("accountID", args[0]);
  message.setInvocationProperty("userID", args[1]);

  // For POST method
  if (args.length > 2) {
   message.setPayload(args[2]);
  }

  return message;
 }
}

We have everything in place to make it work. One thing I don't like is that Jersey for requests with URL not met is returning 404 HTTP status (Not Found). I find 400 status (Bad Request) more appropriate in such case and I'd like to keep 404 reserved for situations where URL was met, but resource was not found. Satysfying that requirement is fairly simple. We need to add a custom exception mapper in Jersey definition:
<jersey:exception-mapper class="pl.poznachowski.jerseyrest.BadURIExceptionMapper" />
with implementation:
public class BadURIExceptionMapper implements ExceptionMapper<NotFoundException> {

  public Response toResponse(NotFoundException exception){
      return Response.status(Response.Status.BAD_REQUEST).entity("Unknown resource: " + exception.getNotFoundUri().toString()).build();
  }
}

Note: It is possible to catch exceptions thrown in the binded outbound endpoints. To make that possible we just need to declare exception in binding interface method.

To test described solution I used the same set of tests as in previous post. However I was having problems testing successful scenarios using MUnit:
ERROR 2013-10-17 22:30:02,434 [main] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message               : Failed to invoke JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Component that caused exception is: JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Message payload is of type: String
Code                  : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. The required object/property "muleContext" is null (java.lang.IllegalArgumentException)
  org.mule.DefaultMuleMessage:292 (null)
2. Failed to invoke JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Component that caused exception is: JerseyResourcesComponent$$EnhancerByCGLIB$$8a425d6{JerseyRestFlow.component.206873183}. Message payload is of type: String (org.mule.component.ComponentException)
  org.mule.component.AbstractComponent:148 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/component/ComponentException.html)

It seems that muleContext is not propagated correctly with MUnit and reflective proxy classes. Switching test class to use 'official' FunctionalTestCase instead of MUnit worked like a charm.

Full example at GitHub: JerseyREST

That were all the solutions for exposing REST Services I can think of.
In my work project, I'm using the REST Module and it's doing its job well. Thus, if you want build REST API and use all of the nifty Mule features  I would suggest going with the Router Module.
If you prefer to be more Java-centric, make use of Mule for handling integration matter only or to use Mule flows to a smaller extent then Jersey approach will fit in.

For more information about Component Bindings I encourage you to read Mule's well-written blog post explaining the feature.

Friday, October 11, 2013

Exposing RESTful interface with Mule pt.1

Update (20.10.2013) - When not supported HTTP method is used, return more appropriate HTTP status (405 instead of 400).

As it's my first blog entry I'd like to welcome everyone. If you'd like to know more about me click on the tab above. If not.. I'll go straight to the point.

Recently, I had to create Mule's application, which exposes itself via a simple RESTful API. However, when it comes to REST, Mule ESB seems to be quite limited in viable options.
The only, official approach is to use the REST Component, which relies on Jersey, which is the Reference Implementation of JAX-RS. Sounds good, but it ties you to the Java code instead of having fun with the Mule's message processing :) This can be overcome as well, but still some Java needs to be written.

Fortunately, there are some other possibilites, which would make simple REST API creation easier. Nevertheless, I'd like to present you all the options (of which I'm aware of) with description and a working example.

All presented examples were tested against Mule ESB 3.4.0 EE. However, they should run without any problems on CE and even on previous versions.

1. "Poor man's" REST:

First approach is the most straightforward one - handling HTTP properties available in Mule manually. Quick overview over HTTP properties is available here - (1.4.7 Use Case) in archived Mule Developer Resources.

In that case, we need to take care of detecting the HTTP method, URL and fetching URL variable parameters by ourselves. We can accomplish that using Mule's choice component and setting flow variables (if needed).
An example will accept URL (on 8088 port) as follows: /client/{accountID}/{userID}/get on both GET and POST HTTP methods, where {accountID} and {userID} are the variable parameters.

General overview of the flow:


First step is not necessary, but helpful for quick suppresion of favicon.ico requests when accesing service via browsers:
<message-filter doc:name="Filter favicon">
  <not-filter>
    <wildcard-filter casesensitive="true" pattern="/favicon.ico">
  </wildcard-filter></not-filter>
</message-filter>
To match URL let's use choice component with regular expression:
<choice doc:name="Choice">
  <when expression="#[regex('/client/\\w+/\\w+/get/?', message.inboundProperties['http.request.path'])]">
   ...
 </when>
  <otherwise>
    <http:response-builder status="400" doc:name="Return 400 For bad URL">
      <set-payload value="Unknown resource: #[message.inboundProperties['http.request.path']]" />
    </http:response-builder>
  </otherwise>
</choice>
We retrieve URL path using Mule's inbound property - 'http.request.path', then match it with the regexp '/client/\\w+/\\w+/get/?'. If we don't have a match we just simply return 400 HTTP status.
Otherwise, we can proceed and determine which HTTP method was used to send the request:
<choice doc:name="Choice">
  <when expression="message.inboundProperties['http.method'] == 'GET'">
    <flow-ref name="RetrievingParameters" doc:name="Retrieve Parameters"/>
    <flow-ref name="ProcessGET" doc:name="Process GET"/>
  </when>
  <when expression="message.inboundProperties['http.method'] == 'POST'">
    <flow-ref name="RetrievingParameters" doc:name="Retrieve Parameters"/>
    <flow-ref name="ProcessPOST" doc:name="Process POST"/>
  </when>
  <otherwise>
    <http:response-builder status="405" doc:name="Return 405 For bad HTTP method">
      <set-payload value="Unknown HTTP method: #[message.inboundProperties['http.method']]" />
    </http:response-builder>
  </otherwise>
</choice>
This time we are checking another inbound parameter - 'http.method'. If it's not the method we want to proceed with, we return 405 HTTP status (Method Not Allowed).

For all supported methods we want to fetch the URL parameters. Hence, this functionality is extracted to a reusable sub-flow:
<sub-flow name="RetrievingParameters" doc:name="RetrievingParameters">
  <set-variable variableName="accountID" value="#[StringUtils.splitAndTrim(message.inboundProperties['http.request.path'], '/')[1]]" doc:name="Set Account ID"/>
  <set-variable variableName="userID" value="#[StringUtils.splitAndTrim(message.inboundProperties['http.request.path'], '/')[2]]" doc:name="Set User ID"/>
</sub-flow>
As I'm not good at regular expressions, I decided to go with Mule's StringUtils method, which splits the URL and gets me the parameters I'm looking for.

To avoid providing fully qualified name of the class over and over again, we can define global import for Mule expressions:
<configuration doc:name="Configuration">
  <expression-language>
    <import class="org.mule.util.StringUtils" />
  </expression-language>
</configuration>
That is basically everything we needed. We can now process GET / POST requests. For readability and maintainability reasons we will do that in separate flows:
<flow name="ProcessPOST" doc:name="ProcessPOST">
  <http:response-builder status="200"doc:name="Return OK">
    <set-payload value="Processing POST with account id: #[accountID] and user id: #[userID] and body: #[payload]" doc:name="Set Payload"/>
  </http:response-builder>
</flow>
<flow name="ProcessGET" doc:name="ProcessGET">
  <http:response-builder status="200"doc:name="Return OK">
    <set-payload value="Processing GET with account id: #[accountID] and user id: #[userID]" doc:name="Set Payload"/>
  </http:response-builder>
</flow>
 
To test my service I've prepared a test class with couple of Mule functional tests. Instead of using default FunctionalTestCases, I suggest using MUnit. From what I know, it is supposed to replace current Mule testing solution in future. With MUnit you can write your tests using Java (JUnit) or XML (Mule code). It has features like mocking endpoints, processors and ability to call flow directly, skipping inbound endpoints. However, development of MUnit seemed to slow down recently.
To keep the test concise, I used JUnitParams extension, as I wanted to use same test methods, but with different input (URLs).

Additionally, we could've checked for content-type etc., but I didn't want to clutter the code with insignificant details.
Having in mind next example, I'm not exactly sure, if anyone would opt for this approach. But still, there it is, probably for demonstration purposes only ;)

Full example at GitHub: PoorMansREST

2. REST Router

To make our lives easier Mule REST Router Module has been developed. Code is available at GitHub.
Almost everything we did in previous approach is wrapped into one message processor. Installation and configuration is well explained in the links provided.

Flow overview:

Basically, to implement same behaviour, we need to configure rest-router as follows:
<rest-router:router templateUri="/client/{accountID}/{userID}/get">
  <rest-router:get>
    <flow-ref name="ProcessGET" doc:name="Process GET" />
  </rest-router:get>
  <rest-router:post>
    <flow-ref name="ProcessPOST" doc:name="Process POST" />
  </rest-router:post>
</rest-router:router>

REST Router automatically assign parameters specified in templateUri to flow variables. However, I found out that it passes through empty parameters, so it is advisable to validate them explicitly in your code.
Outstanding part is to cover invalid URLs and return something meaningful. For requests not matching our template we return standard 400 HTTP status (placed just after </rest-router:router>) :

<http:response-builder status="400" doc:name="Return 400 For bad URL">
  <set-payload value="Unknown resource: #[message.inboundProperties['http.request.path']]" />
</http:response-builder>

For HTTP methods not supported REST Router should (per documentation) throw UnsupportedHttpVerbException. However, there is an issue I submitted here. Unfortunately, the project looks abandoned as nobody cared :) I did some more research and it appears that the issue is not in the REST Router itself, but how the DevKit java code is generated.
There are two solutions for that:
1) Always implement whole set of methods in REST Router and return appriopriate message / status / throw exception in those you don't want to support.
2) As I don't like to have redundancy in my code I used a dirty hack of replacing compiled class with a fixed one directly in the jar file. Amended version of the module is available here. Just install it in your local Maven repository and switch module version in your pom.xml file to 1.2-fixed.

Fix description:
HTTP method recognition is implemented as in the following snippet: (RestRouterModule.java)
if (get != null && method.equalsIgnoreCase("get")) {
 return get.processWithExtraProperties(properties);
} else if (put != null && method.equalsIgnoreCase("put")) {
 return put.processWithExtraProperties(properties);
} else if (post != null && method.equalsIgnoreCase("post")) {
 return post.processWithExtraProperties(properties);
} else if (delete != null && method.equalsIgnoreCase("delete")) {
 return delete.processWithExtraProperties(properties);
} else if (patch != null && method.equalsIgnoreCase("patch")) {
 return patch.processWithExtraProperties(properties);
} else {
 throw new UnsupportedHttpVerbException(method);
}
Variables: get, put, post... are instances of NestedProcessor class, marked @Optional.
However, code generated by DevKit always propagates those variables to the RestRouterModule as instances of NestedProcessorChain (RouterMessageProcessor.java):
final NestedProcessor _transformedGet = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) get));
final NestedProcessor _transformedPut = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) put));
final NestedProcessor _transformedPost = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) post));
final NestedProcessor _transformedDelete = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) delete));
final NestedProcessor _transformedPatch = new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) patch));
Those _transformed method variables are passed into the RestRouterModule class, hence it always tries to process the processWithExtraProperties(properties) method and eventually finishes with NullPointerException as the last argument in NestedProcessorChain contructor is null.

My fixed version of RouterMessageProcessor.java includes following change:
final NestedProcessor _transformedGet = (get == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) get));
final NestedProcessor _transformedPut = (put == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) put));
final NestedProcessor _transformedPost = (post == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) post));
final NestedProcessor _transformedDelete = (delete == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) delete));
final NestedProcessor _transformedPatch = (patch == null) ? null : new NestedProcessorChain(event, getMuleContext(), ((MessageProcessor) patch));
Not much to explain here. Now, if a REST module HTTP method is not defined it will propagate null to RestRouterModule.java and throw UnsupportedHttpVerbException as expected.

In the flow we use standard choice exception strategy and catch the UnsupportedHttpVerbException to return 405 HTTP status:

  
    
      
    
  


For testing I used the same set of tests as in previous example. As you can see, it's much quicker and cleaner approach that saves you a lot of time and lines of code :)

Full example (using modified version of the module) at GitHub: RestWithRouter

In the next part I will present how to achieve the same, but using Jersey with writing as little Java as possible. Go here for the second part!