Smooks Data Integration

Bård (Langöy) and I have been doing lots of work on improving our EDI support, specifically in the area of handling UN/EDIFACT (WikipediA) message Interchanges.  The following improvements are now in the Smooks v1.4 codebase (available from the 1.4-SNAPSHOT):

1. A new UN/EDIFACT Interchange Reader (<unedifact:reader>), which allows Smooks to process UN/EFIFACT message Interchanges (one or more UN/EDIFACT messages wrapped in the UN/EDIFACT Interchange control segments).
2. A new Maven/Ant tool called the “EDI Conversion Tool” (ECT) that can take the official UN/EDIFACT message definition directory zip files and from them, generate a jar file containing a set of equivalent Smooks EDI Mapping Models.  This tool is very easy to configure and use, but we are in the processing of pre-generating a library of these jars for all the UN/EDIFACT message sets and making them available trough the central maven repository, making it even easier to consume UN/EDIFACT messages with Smooks.  We hope to add support for other formats (X.12, HL7 etc) in future releases of Smooks.

So that’s what’s available in the current 1.4 codebase (1.4-SNAPSHOT), but we’re also in the process of making more additions to this for Smooks v1.4.  We’re extending the EDI Java Compiler (EJC) tools to support Java Model generation from the EDI Mapping Model sets generated by the ECT tool (mentioned above).  We’ll also make these available as pre-built jar files from the Maven repository.

Another improvement we’ve made in the 1.4 codebase is the addition of support for Java Model to EDI serialization on the EJC generated Java object models, meaning we’ll be able to do full round trip binding of an EDI (and UN/EDIFACT) message to a populated Java Object model that can be modified and then serialize back out to an EDI stream.  Or, your app can “manually” construct the same Java Object models and serialize them out to an EDI Stream (think JAXB).

Generating a Mapping Model Set using ECT

The easiest way to consume UN/EDIFACT messages using Smooks is to use ECT (EDI Conversion Tool) to generate the EDI Mapping Model set (if we haven’t already pre-built them).  The steps are really easy:

1. Download the official UN/EDIFACT message definitions directory zip file you are interested in from the UNECE website.
2. Create a maven module for the EDI Mapping Model set to be generated and add the maven-ect-plugin (see below) to it’s POM.
3. Execute “mvn clean install” in your maven module and the EDI Mapping Model set jar file will be generated as normal in the modules target folder, and installed into your local maven repository.

The following is an example of the maven module POM for generating the EDI Mapping Model set jar for the d03b.zip definitions directory zip file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.milyn.edi.unedifact</groupId>
    <artifactId>d03b-mapping</artifactId>
    <version>1.0.SNAPSHOT</version>
    <name>UN/EDIFACT - D03B - Mapping Model</name>

    <build>
        <plugins>
            <plugin>
                <groupId>org.milyn</groupId>
                <artifactId>maven-ect-plugin</artifactId>
		        <version>1.4-SNAPSHOT</version>
                <configuration>
                    <src>d03b.zip</src>
                    <srcType>UNEDIFACT</srcType>
                </configuration>
                <executions>
                    <execution>
                        <goals><goal>generate</goal>
                    </goals></execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

Of course when we execute this step, we will perform “mvn clean deploy” and install the jar file in the public maven repository, making it publicly available and thereby removing the need for you to perform this step.  A peek into the public SNAPSHOT repo shows that we’ve started this process.

Using a Mapping Model Set on the UN/EDIFACT Reader

Using a generated EDI Mapping Model set jar in your application (via Smooks) is trivial.  You need to:

1. Add the generated EDI Mapping Model set jar to your classpath (e.g. using a maven dependency configuration).
2. Add the <unedifact:reader> configuration to your Smooks configuration, using the generated jar’s maven groupId, artifactId and version to reference that particular mapping model set through a URN (see below).

An example of the <unedifact:reader> configuration for the above generated EDI Mapping Model set jar for the d03b.zip definitions directory zip file would be as follows:

<unedifact:reader
     mappingModel="urn:org.milyn.edi.unedifact:d03b-mapping:1.0-SNAPSHOT" />

Using the “urn:<groupId>:<artifactId>:<version>” URN pattern, Smooks can find the EDI Mapping Model set jar file on the classpath,

And that’s about it.  The associated Smooks instance will accept a UN/EDIFACT message interchange containing one or more messages defined in the d03b.zip definitions directory zip file, and will generate a stream of SAX events into Smooks, which can be processed using all the other Smooks capabilities (Java Binding, Validation, Templating etc).  As stated earlier, the improvements we are working on for EJC will result in us being able to generate Java Object models from an EDI Mapping Model set jar.

Play with the Examples… and please give feedback

Check out and build the Smooks examples from https://svn.codehaus.org/milyn/trunk/smooks-examples (mvn install).

Change into the “ect-unedifact” directory after building all the examples (must build them all so as to install the poms) and run the example using mvn exec:java. The ect-unedifact example depends on the “d03b-mapping” module which has been pre-built and is available in the public SNAPSHOT repo.

And guys…. PLEASE let us know how you get on with this stuff… things you like and dislike etc !!!

We were slow about doing it, but eventually… Smooks v1.3 (final) has hit the shelves!!

Download and try it out.

Thanks to everyone that contributed to this release.  As well as 65+ bug fixes, we’ve added a number of new features and improvements:

1. Greatly improved XPath support in selectors. This allows you to write more complex selectors with XPath syntax. See here. See JIRA.
2. Support for Fixed Length Field (FLF) messages. See JIRA.
3. Java Binding Improvements:
   • Ability to bind raw data values into the bean context. See JIRA.
   • Support for factory methods for bean creation. See JIRA.
   • Make bind data available to Expression Base Bindings. See JIRA.
4. String manipulation functions for tweaking CSV and FLF fields before forwarding to Smooks components. See JIRA.
5. Support premature Filter termination. See JIRA.
6. Simple fragment splitter. See JIRA.
7. CSV Header Validation. See JIRA.

See here for a full list of Fixes and Features

I did a little playing with Camel over the weekend… had a go at integrating Smooks.

I think those using Camel may find it interesting as another option.  The integration was simple enough and can be seen in this svn workspace (including tests etc).  With better knowledge of Camel, I’m sure there’s a better way of integrating it, but at least it’s a start.

At the moment, you can map String or Java results into the out-message of the Camel Exchange, so you can do templating or java binding (not exposing validation results yet) of XML and non-XML data.  So you can do lots of the normal Smooks type processing.

The aspect I think most Camel users might find interesting is the potential wrt splitting and routing of huge messages (XML and non-XML).  Basically… being able to split and route huge messages as you stream them i.e. not the two step method that seems to be there at the moment – 1) split out all messages 2) route all messages.

Lets look at a simple example, where we have an XML message stream containing customer account activity logs (or… could be CSV formatted).  We want to route each log event to JMS queues based on nationality.

The RouteBuilder config (can also configure the SmooksProcessor via Java ala DSL) …

public class MyRouteBuilder extends RouteBuilder {

    public void configure() throws Exception {

        from("file:src/data/logs?noop=true").
            process(new SmooksProcessor("/configs/log-split.xml", this));

        from("jms:queue:ireland").process(new LogProcessor("ie"));
        from("jms:queue:greatbritain").process(new LogProcessor("gb"));
    }
}

And then the Smooks “log-split.xml” config…

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
          xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.3.xsd"
          xmlns:camel="http://www.milyn.org/xsd/smooks/camel-1.4.xsd">

    <!-- Create a new ActivityLogEvent for each log entry... -->
    <jb:bean beanId="logEvent" class="com.acme.activity.ActivityLogEvent"
                                  createOnElement="activity/event">
        <jb:value property="transactionId" data="event/@txId" />
        <jb:value property="customerId" data="event/customer/@customerId" />
        <jb:value property="nationality" data="event/customer/countryCode" />
        <!-- etc... -->
    </jb:bean>

    <!-- Route GB based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:greatbritain"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'GB'</condition>
    </camel:router>

    <!-- Route Irish based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:ireland"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'IE'</condition>
    </camel:router>

</smooks-resource-list>

The <jb:bean> config creates an ActivityLogEvent bean instance for every activity log. The following <camel:router> configs conditionally route the ActivityLogEvent bean instances to the specified JMS endpoints based on the country-code on the event log.  Of course, using Smooks Templating, you could apply a template to the ActivityLogEvent to produce some other data format and route that instead (a simple template generated String bean bound into the Smooks BeanContext).

The size of the event log is irrelevant here – could be GBs.  Only one ActivityLogEvent instance is in memory at any given time (a new instance created for each event).  The routing is performed as the event log is being streamed i.e. not batched up for routing in a later step.  Assumption here is that there’s a way of telling the Camel File component to forward the file as a stream/reader.

The current implementation is available in SVN here.  Since I’m not overly familiar with Camel, I’m sure there are things we could do better/different.  Please let us know!!!

At JBoss, we’ve been making some good progress with the JBoss Tools Smooks Editor.  The following are two rough Screencasts illustrating two use cases that extend from each other.

EDI to Java

In this Screencast, we bind EDI Purchase Order data (but can also be XML, JSON etc) into an “Order” object model (Java).  This Screencast was created a few weeks ago, using the v3.1.0.CR1 release of JBoss Tools.

To enlarge, press screen icon (bottom right)

EDI to XML (via XSD)

In this Screencast, we extend the example shown in the above Screencast, applying a template to the “Order” object model to produce an XML Purchase Order.  The XML Purchase Order is defined in an XML Schema (XSD), but you’ll see how the editor makes it very easy to work with the XSD.  This Screencast was created recently, using one of the Nightly Builds of JBoss Tools.

To enlarge, press screen icon (bottom right)

Status

We are currently working towards the v3.1.0.CR2 release of JBoss Tools, which is due to be released in Feb (not sure of exact date).  There are still a number of issues with the editor.  We are working on these and will be resolving many of them for the 3.1.0.CR2 release.

We have purposely restricted the 3.1.0 version of the editor:

1. Supported Use Cases:
   • Java Mapping/Binding:  Drag & Drop mapping/binding of XML, JSON, CSV and EDI data to Java.
   • CSV and XML Templating:  Drag & Drop templating of Java Mappings (see #1 above) to produce CSV or XML (XML via XSD or XML Sample).
2. Supported Integration:
   • JBoss Tools ESB Editor: Will have basic open and edit integration with the ESB SmooksAction.
3. Supported Versions:
   • Smooks v1.2 Configurations:  The Editor will only support Smooks v1.2 configuration namespaces.  Therefore, if you have configurations using config namespaces from earlier (or newer v1.3) versions of Smooks, the editor will not open the configurations.

So, the 3.1.0 version of the editor will not support:

1. Direct Source to Result drag & drop templating e.g. XML to XML, EDI to XML, XML to CSV etc.  We hope to support this in a future release (next one hopefully), but for now (in the v3.1.0 release), you need to define a Java Object model to use as the canonical form of the data being mapped i.e. v3.1.0 templating works as Source -> Java -> Result.
2. A number of features supported by the Smooks Runtime, such as Persistence, Validation, Splitting & Routing etc.  We will be adding support for these in future releases.
3. Smooks v1.1 (and before) configuration namespaces that have newer versions in Smooks v1.2.  For example, Smooks v1.2 added a v1.2 javabean config namespace.  It is this version of the javabean config namespace that is supported by the editor.  The v1.1 javabean config namespace is not supported.
4. Smooks v1.3 (yet to be released) configuration namespaces.

Try it Out…

Please take the editor for a spin and let us know what you think.  It can be downloaded (standalone, or as part of JBoss Tools) from the JBoss Tools Downloads Page.  Please log any issues you find with the editor (against the “smooks” component), if not already logged.

We’ve just made a Smooks v1.2.4 patch release (Download).

This contains:

1. Some performance enhancements (reader pooling),
2. Some bug fixes and,
3. One small new feature to support Basic Fragment Splitting and Routing.  Right… we normally only do fixes in patch releases, but I wanted to get this into the JBoss SOA Platform before it released, so we made an exception for this one.

I would like to introduce the new factory feature in the Javabean cartridge of the up coming Smooks 1.3 release. This new features makes it possible to use a static factory method or a factory object to instantiate objects with the Javabean cartridge.

Continue reading…

Of late, JBoss have been making more of a push on their Smooks Eclipse Tooling within JBoss Tools.  Prior to a few weeks ago, Dart Peng was “more or less” left to his own devices on this work, without much help from myself or anyone else.  More recently, Brian Fitzpatrick (JBoss) and myself have been more actively contributing to the great work being done by Dart.

Brian has been blogging about this work here on the JBoss Tools blog. Please take a look and contribute your throughs and ideas!!!

SOPERA have created a new Open Source project on Google called “soperadi-smooks“.  It is (or is hopefully going to be) a Smooks component for Talend.

Cool!!!!

Smooks can produce “output” in a number of different ways.  It can produce multiple sets of results, as well as output fragments of data to multiple different endpoints (ESB Service Endpoints, Files, JMS Destinations, DBs etc… and in different formats), all in a single filtering pass of a message stream.

For more on this, please see the Smooks Wiki…

Smooks v1.2 has just been released (download).  This release adds a number of very interesting features and enhancements to Smooks, most notably:

1. EDIFACT Java Compiler (EJC):  EJC greatly simplifies the process of binding EDI data to a Java object graph.  EJC is similar to JAXBs XJC, accept for EDI messages.  This is just the first of a number of developer optimization features we are in the process of adding in the area of EDI message processing.
2. Entity Persistence Framework Support:  Reuse your Entity Persistence resources (like Hibernate, iBATIS or any JPA compatible resource) to persist or enrich messages of any format (EDI, XML etc).  Access a database and use it’s query language, or CRUD methods, for reading and writing to the database.
3. Validation:  Perform Rule based message fragment validation on messages of any format (EDI, XML etc).  Supports Data Field level validation using regular expressions, or Business Rule compliance validation using MVEL expressions.
4. Simplified CSV Processing:  Bind CSV records to Java objects in seconds using a much simplified XML configuration, or using a very simple programmatic API.
5. Improved Programmatic APIs:  Prior to Smooks v1.2, programmatic configuration was not one of Smooks’ strengths.   In Smooks v1.2 we’ve made significant improvements in this area.  Most Smooks features can now be utilised through Java, without writing a line of XML.

As well as these new features, Smooks v1.2 includes numerous bug fixes.  Please download Smooks v1.2 and let us know what you think.  We hope you find it useful!!