Smooks Data Integration

Renat Zubairov has been doing some really cool stuff with Smooks UN/EDIFACT.  On top of the ECT tooling, he has built tools for generating Eclipse EMF Ecore (XMI) models for each of the EDI Mapping Models being produced by ECT.  On the back of this work, he has been able to build an Editor for Eclipse, allowing you to read and write UN/EDIFACT messages from inside Eclipse.

See more here on the Smooks Wiki.

The UN/EDIFACT Editor for Eclipse is cool, but I think the wider impact of these capabilities is even cooler.  This is really important work because it will open a number of really important doors in terms of our ability to process and transform not just UN/EDIFACT, but EDI in general (X.12, HL7 etc).  Our long-term goal is to add support for X.12, HL7 etc to ECT (i.e. build conversion tools for these standards).  Once we do this, we will be able to provide out-of-the-box support for these EDI formats on the Smooks Runtime.  We hope it will also mean that we can use this work that Renat is doing to automate building of Ecore/XMI models for these same formats.  Once we have these Ecore/XMI models, we hope we will be able to:

1. Build and provide XML Schemas (XSDs) for all of these messages.
2. Make use of Eclipse EMF Compare to compare models e.g. compare 2 different versions (D93 and D96) of a UN/EDIFACT Invoice specification.
3. Make use of Eclipse EMF M2M to create a transformation model between 2 different models e.g. between UN/EDIFACT and X.12 Invoices.

This is awesome stuff.  Keep up the great work Renat!!!

Here is a quick overview where you can find the slides presented on the Smooks Community Day 2010:

• UN/EDIFACT processing with Smooks
• Reusing existing data layer to persist multi format data streams
• Smooks Sopera DI Components

Bård (Langöy) and I have been doing lots of work on improving our EDI support, specifically in the area of handling UN/EDIFACT (WikipediA) message Interchanges.  The following improvements are now in the Smooks v1.4 codebase (available from the 1.4-SNAPSHOT):

1. A new UN/EDIFACT Interchange Reader (<unedifact:reader>), which allows Smooks to process UN/EFIFACT message Interchanges (one or more UN/EDIFACT messages wrapped in the UN/EDIFACT Interchange control segments).
2. A new Maven/Ant tool called the “EDI Conversion Tool” (ECT) that can take the official UN/EDIFACT message definition directory zip files and from them, generate a jar file containing a set of equivalent Smooks EDI Mapping Models.  This tool is very easy to configure and use, but we are in the processing of pre-generating a library of these jars for all the UN/EDIFACT message sets and making them available trough the central maven repository, making it even easier to consume UN/EDIFACT messages with Smooks.  We hope to add support for other formats (X.12, HL7 etc) in future releases of Smooks.

So that’s what’s available in the current 1.4 codebase (1.4-SNAPSHOT), but we’re also in the process of making more additions to this for Smooks v1.4.  We’re extending the EDI Java Compiler (EJC) tools to support Java Model generation from the EDI Mapping Model sets generated by the ECT tool (mentioned above).  We’ll also make these available as pre-built jar files from the Maven repository.

Another improvement we’ve made in the 1.4 codebase is the addition of support for Java Model to EDI serialization on the EJC generated Java object models, meaning we’ll be able to do full round trip binding of an EDI (and UN/EDIFACT) message to a populated Java Object model that can be modified and then serialize back out to an EDI stream.  Or, your app can “manually” construct the same Java Object models and serialize them out to an EDI Stream (think JAXB).

Generating a Mapping Model Set using ECT

The easiest way to consume UN/EDIFACT messages using Smooks is to use ECT (EDI Conversion Tool) to generate the EDI Mapping Model set (if we haven’t already pre-built them).  The steps are really easy:

1. Download the official UN/EDIFACT message definitions directory zip file you are interested in from the UNECE website.
2. Create a maven module for the EDI Mapping Model set to be generated and add the maven-ect-plugin (see below) to it’s POM.
3. Execute “mvn clean install” in your maven module and the EDI Mapping Model set jar file will be generated as normal in the modules target folder, and installed into your local maven repository.

The following is an example of the maven module POM for generating the EDI Mapping Model set jar for the d03b.zip definitions directory zip file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.milyn.edi.unedifact</groupId>
    <artifactId>d03b-mapping</artifactId>
    <version>1.0.SNAPSHOT</version>
    <name>UN/EDIFACT - D03B - Mapping Model</name>

    <build>
        <plugins>
            <plugin>
                <groupId>org.milyn</groupId>
                <artifactId>maven-ect-plugin</artifactId>
		        <version>1.4-SNAPSHOT</version>
                <configuration>
                    <src>d03b.zip</src>
                    <srcType>UNEDIFACT</srcType>
                </configuration>
                <executions>
                    <execution>
                        <goals><goal>generate</goal>
                    </goals></execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

Of course when we execute this step, we will perform “mvn clean deploy” and install the jar file in the public maven repository, making it publicly available and thereby removing the need for you to perform this step.  A peek into the public SNAPSHOT repo shows that we’ve started this process.

Using a Mapping Model Set on the UN/EDIFACT Reader

Using a generated EDI Mapping Model set jar in your application (via Smooks) is trivial.  You need to:

1. Add the generated EDI Mapping Model set jar to your classpath (e.g. using a maven dependency configuration).
2. Add the <unedifact:reader> configuration to your Smooks configuration, using the generated jar’s maven groupId, artifactId and version to reference that particular mapping model set through a URN (see below).

An example of the <unedifact:reader> configuration for the above generated EDI Mapping Model set jar for the d03b.zip definitions directory zip file would be as follows:

<unedifact:reader
     mappingModel="urn:org.milyn.edi.unedifact:d03b-mapping:1.0-SNAPSHOT" />

Using the “urn:<groupId>:<artifactId>:<version>” URN pattern, Smooks can find the EDI Mapping Model set jar file on the classpath,

And that’s about it.  The associated Smooks instance will accept a UN/EDIFACT message interchange containing one or more messages defined in the d03b.zip definitions directory zip file, and will generate a stream of SAX events into Smooks, which can be processed using all the other Smooks capabilities (Java Binding, Validation, Templating etc).  As stated earlier, the improvements we are working on for EJC will result in us being able to generate Java Object models from an EDI Mapping Model set jar.

Play with the Examples… and please give feedback

Check out and build the Smooks examples from https://svn.codehaus.org/milyn/trunk/smooks-examples (mvn install).

Change into the “ect-unedifact” directory after building all the examples (must build them all so as to install the poms) and run the example using mvn exec:java. The ect-unedifact example depends on the “d03b-mapping” module which has been pre-built and is available in the public SNAPSHOT repo.

And guys…. PLEASE let us know how you get on with this stuff… things you like and dislike etc !!!