We were slow about doing it, but eventually… Smooks v1.3 (final) has hit the shelves!!

Download and try it out.

Thanks to everyone that contributed to this release.  As well as 65+ bug fixes, we’ve added a number of new features and improvements:

1. Greatly improved XPath support in selectors. This allows you to write more complex selectors with XPath syntax. See here. See JIRA.
2. Support for Fixed Length Field (FLF) messages. See JIRA.
3. Java Binding Improvements:
   • Ability to bind raw data values into the bean context. See JIRA.
   • Support for factory methods for bean creation. See JIRA.
   • Make bind data available to Expression Base Bindings. See JIRA.
4. String manipulation functions for tweaking CSV and FLF fields before forwarding to Smooks components. See JIRA.
5. Support premature Filter termination. See JIRA.
6. Simple fragment splitter. See JIRA.
7. CSV Header Validation. See JIRA.

See here for a full list of Fixes and Features

I did a little playing with Camel over the weekend… had a go at integrating Smooks.

I think those using Camel may find it interesting as another option.  The integration was simple enough and can be seen in this svn workspace (including tests etc).  With better knowledge of Camel, I’m sure there’s a better way of integrating it, but at least it’s a start.

At the moment, you can map String or Java results into the out-message of the Camel Exchange, so you can do templating or java binding (not exposing validation results yet) of XML and non-XML data.  So you can do lots of the normal Smooks type processing.

The aspect I think most Camel users might find interesting is the potential wrt splitting and routing of huge messages (XML and non-XML).  Basically… being able to split and route huge messages as you stream them i.e. not the two step method that seems to be there at the moment – 1) split out all messages 2) route all messages.

Lets look at a simple example, where we have an XML message stream containing customer account activity logs (or… could be CSV formatted).  We want to route each log event to JMS queues based on nationality.

The RouteBuilder config (can also configure the SmooksProcessor via Java ala DSL) …

public class MyRouteBuilder extends RouteBuilder {

    public void configure() throws Exception {

        from("file:src/data/logs?noop=true").
            process(new SmooksProcessor("/configs/log-split.xml", this));

        from("jms:queue:ireland").process(new LogProcessor("ie"));
        from("jms:queue:greatbritain").process(new LogProcessor("gb"));
    }
}

And then the Smooks “log-split.xml” config…

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
          xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.3.xsd"
          xmlns:camel="http://www.milyn.org/xsd/smooks/camel-1.4.xsd">

    <!-- Create a new ActivityLogEvent for each log entry... -->
    <jb:bean beanId="logEvent" class="com.acme.activity.ActivityLogEvent"
                                  createOnElement="activity/event">
        <jb:value property="transactionId" data="event/@txId" />
        <jb:value property="customerId" data="event/customer/@customerId" />
        <jb:value property="nationality" data="event/customer/countryCode" />
        <!-- etc... -->
    </jb:bean>

    <!-- Route GB based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:greatbritain"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'GB'</condition>
    </camel:router>

    <!-- Route Irish based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:ireland"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'IE'</condition>
    </camel:router>

</smooks-resource-list>

The <jb:bean> config creates an ActivityLogEvent bean instance for every activity log. The following <camel:router> configs conditionally route the ActivityLogEvent bean instances to the specified JMS endpoints based on the country-code on the event log.  Of course, using Smooks Templating, you could apply a template to the ActivityLogEvent to produce some other data format and route that instead (a simple template generated String bean bound into the Smooks BeanContext).

The size of the event log is irrelevant here – could be GBs.  Only one ActivityLogEvent instance is in memory at any given time (a new instance created for each event).  The routing is performed as the event log is being streamed i.e. not batched up for routing in a later step.  Assumption here is that there’s a way of telling the Camel File component to forward the file as a stream/reader.

The current implementation is available in SVN here.  Since I’m not overly familiar with Camel, I’m sure there are things we could do better/different.  Please let us know!!!

At JBoss, we’ve been making some good progress with the JBoss Tools Smooks Editor.  The following are two rough Screencasts illustrating two use cases that extend from each other.

EDI to Java

In this Screencast, we bind EDI Purchase Order data (but can also be XML, JSON etc) into an “Order” object model (Java).  This Screencast was created a few weeks ago, using the v3.1.0.CR1 release of JBoss Tools.

To enlarge, press screen icon (bottom right)

EDI to XML (via XSD)

In this Screencast, we extend the example shown in the above Screencast, applying a template to the “Order” object model to produce an XML Purchase Order.  The XML Purchase Order is defined in an XML Schema (XSD), but you’ll see how the editor makes it very easy to work with the XSD.  This Screencast was created recently, using one of the Nightly Builds of JBoss Tools.

To enlarge, press screen icon (bottom right)

Status

We are currently working towards the v3.1.0.CR2 release of JBoss Tools, which is due to be released in Feb (not sure of exact date).  There are still a number of issues with the editor.  We are working on these and will be resolving many of them for the 3.1.0.CR2 release.

We have purposely restricted the 3.1.0 version of the editor:

1. Supported Use Cases:
   • Java Mapping/Binding:  Drag & Drop mapping/binding of XML, JSON, CSV and EDI data to Java.
   • CSV and XML Templating:  Drag & Drop templating of Java Mappings (see #1 above) to produce CSV or XML (XML via XSD or XML Sample).
2. Supported Integration:
   • JBoss Tools ESB Editor: Will have basic open and edit integration with the ESB SmooksAction.
3. Supported Versions:
   • Smooks v1.2 Configurations:  The Editor will only support Smooks v1.2 configuration namespaces.  Therefore, if you have configurations using config namespaces from earlier (or newer v1.3) versions of Smooks, the editor will not open the configurations.

So, the 3.1.0 version of the editor will not support:

1. Direct Source to Result drag & drop templating e.g. XML to XML, EDI to XML, XML to CSV etc.  We hope to support this in a future release (next one hopefully), but for now (in the v3.1.0 release), you need to define a Java Object model to use as the canonical form of the data being mapped i.e. v3.1.0 templating works as Source -> Java -> Result.
2. A number of features supported by the Smooks Runtime, such as Persistence, Validation, Splitting & Routing etc.  We will be adding support for these in future releases.
3. Smooks v1.1 (and before) configuration namespaces that have newer versions in Smooks v1.2.  For example, Smooks v1.2 added a v1.2 javabean config namespace.  It is this version of the javabean config namespace that is supported by the editor.  The v1.1 javabean config namespace is not supported.
4. Smooks v1.3 (yet to be released) configuration namespaces.

Try it Out…

Please take the editor for a spin and let us know what you think.  It can be downloaded (standalone, or as part of JBoss Tools) from the JBoss Tools Downloads Page.  Please log any issues you find with the editor (against the “smooks” component), if not already logged.

We’ve just made a Smooks v1.2.4 patch release (Download).

This contains:

1. Some performance enhancements (reader pooling),
2. Some bug fixes and,
3. One small new feature to support Basic Fragment Splitting and Routing.  Right… we normally only do fixes in patch releases, but I wanted to get this into the JBoss SOA Platform before it released, so we made an exception for this one.

I would like to introduce the new factory feature in the Javabean cartridge of the up coming Smooks 1.3 release. This new features makes it possible to use a static factory method or a factory object to instantiate objects with the Javabean cartridge.

Before this new feature you could only instantiate new beans by providing the class name of the bean and the bean would then be created using it’s public parameterless constructor. So it was mandatory to have a parameterless constructor on the Javabean. Here is an example:

<jb:bean
   beanId="orders"
   class="java.util.ArrayList"
   createOnElement="orders"
>
     <!-- ... bindings -->
</jb:bean>

But now it is possible to use a factory to instantiate the bean. So you don’t need a public parameterless constructur anymore. You even don’t have to define the actual class name any more. Any of the interfaces that the class implements suffices. However only the methods of that interface are available for binding then. Here is an example where we instantiate an ArrayList object using a static factory method:

<jb:bean
   beanId="orders"
   class="java.util.List"
   factory="some.package.ListFactory#newList"
   createOnElement="orders"
>
     <!-- ... bindings -->
</jb:bean>

With the factory attribute you set the factory definition. The class attributes defines that a List object is returned. What kind of List object (ArrayList, LinkedList) is up to the ListFactory to decide. Here is another example:

<jb:bean
   beanId="orders"
   class="java.util.List"
   factory="some.package.ListFactory#getInstance.newList"
   createOnElement="orders"
>
     <!-- ... bindings -->
</jb:bean>

Here we define that an instance of the ListFactory needs to be retrieved using the static method getInstance() and that then the newList() method needs to be called on the ListFactory object to create the List object. This construct makes it possible to easily call Singleton Factories.

The default factory definition language looks like this: factory.class#static_method{.instance_method}

It is also possible to use MVEL as the factory definition language. MVEL has some advantages over the basic default definition language, for example you can use objects from the bean context as the factory object or you can call factory methods with parameters. These parameters can be defined within the definition or they can be objects from the bean context. To be able to use MVEL you must set the ‘factory.definition.parser.class’ global parameter to ‘org.milyn.javabean.factory.MVELFactoryDefinitionParser’.

Here is an example using MVEL:

<smooks-resource-list
   xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
   xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.3.xsd">

	<params>
		<param name="factory.definition.parser.class">
			org.milyn.javabean.factory.MVELFactoryDefinitionParser
		</param>
	</params>

    <jb:bean
		beanId="orders"
		class="java.util.List"
		factory="some.package.ListFactory.getInstance().newList()"
		createOnElement="orders"
	>
		<!-- ... bindings -->
    </jb:bean>

</smooks-resource-list>

Maybe you wonder why we don’t use MVEL as the default factory definition language? Currently the performance of the basic definition language and MVEL is about equal. The reason that the basic definition language isn’t faster is because it currently uses reflection to call the factory methods. However there are plans to use byte code generation instead of reflection. This should improve the performance dramatically. If MVEL where the default language then we couldn’t do anything to improve the performance for those people who don’t need any thing more then the basic features that the basic definition language offers.

If you have any questions or remarks then I would gladly hear them.

Of late, JBoss have been making more of a push on their Smooks Eclipse Tooling within JBoss Tools.  Prior to a few weeks ago, Dart Peng was “more or less” left to his own devices on this work, without much help from myself or anyone else.  More recently, Brian Fitzpatrick (JBoss) and myself have been more actively contributing to the great work being done by Dart.

Brian has been blogging about this work here on the JBoss Tools blog. Please take a look and contribute your throughs and ideas!!!

SOPERA have created a new Open Source project on Google called “soperadi-smooks“.  It is (or is hopefully going to be) a Smooks component for Talend.

Cool!!!!

Smooks can produce “output” in a number of different ways.  It can produce multiple sets of results, as well as output fragments of data to multiple different endpoints (ESB Service Endpoints, Files, JMS Destinations, DBs etc… and in different formats), all in a single filtering pass of a message stream.

For more on this, please see the Smooks Wiki…

Smooks v1.2 has just been released (download).  This release adds a number of very interesting features and enhancements to Smooks, most notably:

1. EDIFACT Java Compiler (EJC):  EJC greatly simplifies the process of binding EDI data to a Java object graph.  EJC is similar to JAXBs XJC, accept for EDI messages.  This is just the first of a number of developer optimization features we are in the process of adding in the area of EDI message processing.
2. Entity Persistence Framework Support:  Reuse your Entity Persistence resources (like Hibernate, iBATIS or any JPA compatible resource) to persist or enrich messages of any format (EDI, XML etc).  Access a database and use it’s query language, or CRUD methods, for reading and writing to the database.
3. Validation:  Perform Rule based message fragment validation on messages of any format (EDI, XML etc).  Supports Data Field level validation using regular expressions, or Business Rule compliance validation using MVEL expressions.
4. Simplified CSV Processing:  Bind CSV records to Java objects in seconds using a much simplified XML configuration, or using a very simple programmatic API.
5. Improved Programmatic APIs:  Prior to Smooks v1.2, programmatic configuration was not one of Smooks’ strengths.   In Smooks v1.2 we’ve made significant improvements in this area.  Most Smooks features can now be utilised through Java, without writing a line of XML.

As well as these new features, Smooks v1.2 includes numerous bug fixes.  Please download Smooks v1.2 and let us know what you think.  We hope you find it useful!!

Smooks 1.2 adds support for message data Validation as one of its new features. This new feature allows you to perform strong field and fragment validation on not just XML data, but also on EDI, JSON, CSV etc. It currently supports Regex and MVEL rules bases (Drools to follow). Regex rules allow you to perform low level field format validation, while MVEL rules allow you to perform Business rules validation at a fragment/message level.

This is not intended as an alternative to Schema based validation such as XSD or Relax.  It provides additional layers of validation in areas not covered by these mechanisms.

Examples of validations one might perform on an Order message containing a collection of Order Item:

1. Use a Regex rule to validate that an orderId field follows a predefined format of “Uppercase Char, followed by 5 digits” e.g. “W12345″ or “U54321″.

2. Use an MVEL rule to enforce a business rule by validating that an order item total is not greater than 50.00 i.e. “price * quantity < 50.00″. Of course these can be more complex.

The easiest way to get familiar with the validation features is to look at an example, so here goes.

For the following example assume that we have the xml that looks like this:

<Order>
    <header>
        <orderId>A188127</orderId>
        <username>user1</username>
        <name>
            <firstname>Harry</firstname>
            <lastname>Fletcher</lastname>
        </name>
        <email>harry.fletcher@gmail.</email>
        <state>South Dakota</state>
    </header>
</Order>

An example of a validation configuration could look like this:

<?xml version="1.0"?>
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
                   xmlns:rules="http://www.milyn.org/xsd/smooks/rules-1.0.xsd"
                   xmlns:validation="http://www.milyn.org/xsd/smooks/validation-1.0.xsd">

   <!-- ruleBases used by validation rules... -->
   <rules:ruleBases>
      <rules:ruleBase name="customer"
                    src="rules/customer.properties"
                    provider="org.milyn.rules.regex.RegexProvider"/>
   </rules:ruleBases>

   <!-- email validation rule targeted at the email element in the header... -->
   <validation:rule executeOn="Order/header/email" name="customer.email" onFail="WARN"/>

</smooks-resource-list>

The ‘exeuteOn‘ attribte specifies the fragment or attribute that you want the validation rule to be applied to. In the case of a Regex rule, it specifies a specific field of data (attribute or element text), while in the case of an MVEL rule, it simply identifies the fragment that triggers execution of that rule.

The next thing to notice is the ‘name‘ attribute. This attribute refers to a rule in one of the configured ruleBase. In the case of the example above, we refer to the ‘email‘ rule in the ‘customer‘ ruleBase i.e. ‘customer.email‘. You can define as many ruleBase elements and validation element as you need, but we are only using one of each here to keep this easy to read.

The ‘customer’ ruleBase is handled by the RegexProvider, as defined in the ‘provider’ attribute on the ruleBase configuration.  We can also see that the ‘customer’ ruleBase rules are located in a properties file named ‘rules/customer.properties‘:

# Email address...
email=^[\\w-\\.]+@([\\w-]+\\.)+[\\w-]{2,4}$

Next, lets take a look at how we execute Smooks to capture some validation results in a ValidationResult instance:

final Smooks smooks = new Smooks("smooks-config.xml");
try
{
    // Create an exec context - no profiles....
    final ExecutionContext executionContext = smooks.createExecutionContext();
    final ValidationResult validationResult = new ValidationResult();

    // Filter the input message...
    smooks.filterSource(executionContext, new StringSource(messageIn), validationResult);

    for (OnFailResult result : results.getWarnings())
    {
        System.out.println("\t" + result.getMessage());
    }
}
finally
{
    smooks.close();
}

As you can see, Validation faiures can be reported at different levels of severity.  This is controlled by setting ‘onFail‘ to one of ‘OK‘, ‘WARN‘, ‘ERROR‘, or ‘FATAL‘. Setting onFail to ‘FATAL’ will cause an exception to be thrown and processing to halt.  The other three levels will not generate an exception.  Failures at these lower severity levels can be accessed through the ValidationResult instance supplied to the Smooks.filterSource method.  An example of getting WARN level failures is shown above with the result.getWarnings() call.

Having usable validation reports is important and Smooks validation supports localized messages for validation failures. These messages can be defined in standard Java ResourceBundle files (.properties format). A convention is used here, based on the rule source name (”src“). The validation message bundle base name is derived from the rule source (”src“) by dropping the rule source file extension and appending a “_messages“.

So for our example we could have a file named ‘customer_messages.properties‘ containing our default locale results:

email=ftl:Invalid email address '${ruleResult.text}' at '${path}'.

So calling result.getMessage() for a validation failure on ‘customer.email’ rule will result in the following message:

Invalid email address ‘harry.fletcher@gmail.’ at ‘/Order/header/email’.

And we could also add a message bundle file for Swedish users.  Following the ResourceBundle naming convention, it should be named ‘customer_messages_sv_SE.properties‘ and would contain the Sweish localized messages:

email=ftl:Felaktig epost adress '${ruleResult.text}' i '${path}'.

Calling result.getMessage(new Locale(”sv, “SE”)) will produce:

Felaktig epost adress ‘harry.fletcher@gmail.’ i ‘/Order/header/email’.

We have only shown a portion of the validation features here.  More information can be found in the Smooks User Guide.  An example of the validation features can be found in Validation Example.