Rob Greathouse (a co-worker of mine at JBoss) was recently asking me how he might use Smooks to feed Java Objects to a Drools Fusion CEP Process.  I took the offical Drools Fusion Stock Tick CEP example and modified it to use Smooks.  Check it out…

I’m proud to announce a new release of Eclipse EDI Editor, now based on Smooks 1.5. This version includes following improvements:

• Migrated to the Smooks 1.5 (SNAPSHOT), which mean that if Smooks can parse your EDI file you can be pretty sure that Eclipse EDI Editor can do it too. It is consistent with Smooks 1.5 in reading and writing EDI files.
• Improved rendering of Component nodes in the editor (see screenshot below)
• Dynamically loading smooks mapping schemas from the mapping bundles generated by smooks, which are (with fixed MILYN-600) are acting as Eclipse plugins now.
• Bringing not only ability to edit UN/EDIFACT files but also contribute all XML Schemas for smooks EDI XML to the Eclipse, so you could edit and validate EDI XML.

To install the Smooks Eclipse EDI Editor just point your Eclipse Update Manager to this URL:

http://d1ce9rnyxjnscn.cloudfront.net/edi-editor

Here is a screencast that shows how to install and use EDI Editor. I recommend you to install it into Eclipse Helios SR2 JEE distribution (this way you will save time to download missing plugins).

http://screencast.com/t/kwEId1tSce

Source of the plugin is available on the Github:

https://github.com/zubairov/edieditor

Check it out and let us know what you think.

This is just a short blog about migrating to Gradle. First up is the unedifact module. The following were the immediate benifits as we see them:

• Simpler versioning of the projects which was a hassle previously as we were using Maven inheritance.
• Less files to maintain.
• Easier to work around specific issues since we have access to the Groovy programming language.
• Now we have project dependencies between the modules which means that we no longer have to build in a specific order, for example first install the mapping project before installing the binding project.

Background

The unedifact module is used to generate the mappings and bindings for various unedifact specifications. For each specification there is a subfolder that is named after the spec.

settings.gradle

This file is where we list all the projects that are to be included in this build. As described in the background section each specification is in a subfolder of the root directory unedifact. But these subfolders are them selves not project but contain projects. For example, a spec folder can contain a ‘mapping’ project, a ‘binding’ project, and a ‘test’ project.

We opted to use the following settings.xml to avoid having to manually update this file when new spec are added:

projectDir = rootProject.getProjectDir().getAbsolutePath()
specsPattern = ~/[\w][\d][\d][\w]/

new File(projectDir).eachFileMatch(specsPattern) { specDir ->
    specDir.eachFileMatch(~/(mapping|binding|test)/) { subModule ->
        moduleName = ":${specDir.getName()}/${subModule.getName()}"
        include moduleName
    }
}

build.gradle
This is where the main part of the configuration is done, well at the moment everything is done in this root build.gradle, but there is nothing stopping subprojects from adding their own build.gradle file and overriding or adding functionality. We have configuration parts that are common to all project and subproject and also configuration sections that are specific for the ‘mapping’, ‘binding’, and ‘test’ projects.

There are project dependencies between the ‘mapping’ project and the ‘binding’ project since the ‘binding’ project uses the mappings generated by the ‘mapping’ project.
The ‘test’ project, if it exists, depends on either the ‘mapping’ project only or both the ‘mapping’ and the ‘binding’ project.

To build you can simply run ‘gradle’ from any of the subproject and it will build its dependencies:

> gradle

This will run a default task which we have set to ‘build’, so this is the same as running:

 gradle build

Adding a new specification
To do this need to create a directory named after the specification. Then create the mapping directory and put the specification zip file in the root of that directory.
Run ‘gradle projects’ to see that you newly added project was added:

> gradle projects
------------------------------------------------------------
Root Project - Smooks Unedifact project
------------------------------------------------------------

Root project 'unedifact' - Smooks Unedifact project
+--- Project ':d00a/binding'
+--- Project ':d00a/mapping'
+--- Project ':d00b/binding'
...

Building is done in the same way as described earlier in this post.

Uploading to Nexus
To deploy artifacts first create a gradle.properties in the root project directory:

milynUser=yourUserName
milynPassword=yourPassword

Uploading to nexus is now done like this

> gradle uploadArchives

Conclusion
While I’m still in the process of learning Gradle and Groovy I feel that most things can be worked around by being able to actually write code, and not having to use xml is very refreshing.
One thing that I added was the task completion which worked like a charm. I added the following tasks to it:

tasks='clean compile dists javadoc jar test war build tasks comparePoms install properties projects uploadArchives'

This will enable task completion so that you can simply type the first few letters of a task and it will complete bash will complete the name for you, like you can do with git.

Next step is to migrate the main Smooks code base to Gradle

Great post by Claus Straube: http://www.catify.com/2011/03/29/transforming-and-splitting-huge-edi-files-with-smooks/

Our work on EDI and UN/EDIFACT support for the next Smooks release continiues. Now we have a namespace support in EDI and UN/EDIFACT generated XML events.  See my blog post for more information and examples on how you could use it.

This post is about the work we have done to improve Smooks OSGI integration.

With Smooks 1.4 we added support for OSGI in the form of making the smooks-all.jar into a bundle. This jar contains all the Smooks classes and resources and is a convenience jar that can be used if you don’t want to keep track of all the individual jars that make up Smooks. During this work we also added a few classes to make Smooks work well inside an OSGI container.

For some background and more details about the Smooks-OSGI work please take a look at this wiki page.

Using Smooks in an OSGI Container (with Camel and ServiceMix)
With 1.4 we added Camel support which has had OSGI support for a long time. We added a few basic examples but it turned out that this was not the best solution. The solution in 1.4 required users to explicitly specify the packages the client bundle required and this made maintenance difficult.
Using Smooks in Camel should not require anything more then declaring that the client bundle uses Camel and perhaps Spring. Client bundles now only need to declare that they import packages for Camel, for example:

Manifest-Version: 1.0
Export-Package: example;uses:="org.apache.camel,
    org.springframework.context.support,
    org.springframework.context"
Bundle-Version: 1.0
Bundle-Name: Milyn Smooks Example - Smooks Camel CVS to XML
Bundle-ManifestVersion: 2
Import-Package:
    example,org.apache.camel;version="2.6",
    org.springframework.context;version="1.2.0",
    org.springframework.context.support;version="1.2.0"
Bundle-SymbolicName: milyn-smooks-example-camel-csv-to-xml

Simplified deployment to ServiceMix
ServiceMix users will hopefully be glad to know that we now generate a features.xml file for easy deployment into service mix. A features.xml specifies all the bundles that a feature need and can be installed/uninstalled as a single unit.
For example, to install the Smooks bundle you can use the following commands:

karaf@root> features:addUrl mvn:org.milyn/milyn-smooks-all/1.5-SNAPSHOT/xml/features
karaf@root> features:install smooks


UN/EDIFACT
Smooks has UN/EDIFACT support which includes pre-generated artifacts that are available in maven. As part of the OSGi work, we have modified these artifacts so as to make them OSGi fragment bundles

Users can now install the mappings and binding as fragment bundles and these will attach to a specific version of the Smooks bundle. This is what this looks like running in Apache ServiceMix 4.3.0.1-fuse:

[ 430] [Active ] [ ] [ ] [ 60] Smooks OSGi (1.5.0.SNAPSHOT)
Fragments: 439,441
[ 439] [Resolved ] [ ] [ ] [ 60] Smooks EDI - UN/EDIFACT - D96A - Bindings (1.4)
Hosts: 430
[ 441] [Resolved ] [ ] [ ] [ 60] Smooks EDI - UN/EDIFACT - D96A - Mapping Model (1.4)
Hosts: 430
[ 448] [Active ] [ ] [Started] [ 60] Milyn Smooks Example - Smooks Camel UNEDIFACT to String (1.0)

Above we can see that the binding (id=439) and the mapping (id=441) have been attached to the Smooks bundle (id=420). More fragment bundles can be attached as required and you can attach the same fragment to different version of the Smooks bundle since we specify a version on the Fragment-Host header.

As mentioned earlier in this post you can find more information and details about this work on this wiki page.

We’re glad to announce the release of Smooks v1.4.  v1.4 has been a long time in the making and has a ton of new features and fixes.  I want to thank everyone that contributed to this release (I think I listed all in the above link… sorry if I missed anyone).  Without the contributions of these people, most of the features in Smooks v1.4 would not exist.

As well as 85+ bug fixes, Smooks v1.4 includes the following new highlight features:

• EDI:
  • Out of the box support for UN/EDIFACT Interchange messages. (By: Bård Langoy, Renat Zubairov, Tom)
  • Allow the EDIParser (by explicit configuration) to ignore unmapped fields. (By: Christian Bach)
  • Added EDI Ojbect Serialization to EJC generated POJOs. (By: Tom)
  • Support for UN/EDIFACT UNA Decimal Point specification. (By: Nicolas Blanc)
• A Dynamic Model Builder that allows you to process evolving configuration namespaces. Supports model reading and writing. (By: Tom)
• Support for reading YAML messages. (By: Maurice Zeijen)
• Create an OSGi compatible “smooks-all” module that assembles all smooks jars into a single jar. (By: Daniel Bevenius, Tom)
• Apache Camel Integration. (By: Daniel Bevenius)
• NumberFormat support for Numeric DataDecoders. (By: Tom)
• Support for bean wiring by bean type and bean annotation. (By: Tom)
• Simple event stream manipulation actions for renaming/removing elements/attributes. (By: Tom)
• Restructured and easier to use User Guide. (By: Maurice Zeijen)
• Restructured and easier to use Examples page. (By: Tom)
• … See full list of Fixes and Features

Two of the main new features in Smooks v1.4 are, out of the box support for UN/EDIFACT Interchanges, and Apache Camel integration.

The new UN/EDIFACT support allows you to process UN/EDIFACT Interchanges by either or both of the following methods:

1. Conversion to XML, allowing you to use standard XML tools (or other Smooks extensions) for further processing.
2. Binding the interchange message data into pre-built Java object models, or custom models if you prefer to use your own.

All the required artifacts for reading any of the UN/EDIFACT directories (and converting to XML) are available in Maven, as are all of the pre-built Java Object models for all the messages defined in all of the UN/EDIFACT directories.  To use them, all you need to do is include the appropriate artifacts into your project.  See the User Guide.

The new Apache Camel integration allows you to hook Smooks into your Camel routes as a Camel Component or Camel DataFormat.  Another important part of the Camel integration are the Smooks extensions that allow you to perform inline routing of Smooks BeanContext data to Camel endpoints i.e. you can also process huge (GB+) data streams.

Putting these two new capabilities together allows us to provide some interesting functionality with respect to Splitting and Routing UN/EDIFACT message Interchanges.  We use Smooks to perform the reading, conversion (to XML and/or Java) and splitting of the interchanges and then use Camel to to perform the inline routing of the Java or XML (using the new Camel extensions for Smooks).

We created an example that demonstrates this capability:

• See the Screencast.
• Get the code and try it yourself (the “splitting-camel” example).

At last we’re nearly there with v1.4.   As well as 85+ bug fixes, Smooks v1.4 includes the following new highlight features:

• Out of the box support for UN/EDIFACT Interchange message.  See Examples.
• Apache Camel Integration.  See Examples.
• A Dynamic Model Builder that’s very handy for dealing with complex evolving configuration namespaces on top of a single Java Object Model. Supports model reading and writing.  See Examples.

But that’s not all by any means.  See here for a full list of Fixes and Features

Download

This entry is about the work we have done integrating Smooks with Apache Camel for Smooks v1.4.  With this tighter integration of these frameworks, you can leverage the power of Smooks to process/transform a wide range of data types, with the power of Apache Camel to route data to a huge range of endpoint types.

Tom started this work a long time ago and got feedback from Claus Ibsen, which Tom implemented. Later Christian Mueller improved the code and I’ve continued working on that code base.

When it comes to using Smooks in Camel there are three options of usage:

1. SmooksComponent
The SmooksComponent is a Camel component which can be used in a Camel route like this:

from("file://inputDir?noop=true")
.to("smooks://smooks-config.xml")
.to("jms:queue:order")

This route will read files from the ‘inputDir’ directory and pass the contents of those files to the Smooks Component. The file will be automatically converted into an instance of javax.xml.transform.stream.StreamSource. TypeConversion is a built-in Camel feature that you can read more about here.

So we know what the source type is but how about the Result type we want Smooks to produce? If you have been following the latest blog posts you might have already guessed that the type(s) are configured in ’smooks-config.xml’ as described in this blog post.

Using the SmooksComponent you can use all the features from Smooks including routing to Camel endpoints. This enables fragments to be routed during transformation. This means that the routing can be performed as the message is being filtered Vs as a batch operation after filtering the message, making it easier to handle large/huge message streams.

Here is an example of routing to camel endpoints using Smooks:

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
         xmlns:camel="http://www.milyn.org/xsd/smooks/camel-1.4.xsd">

  <!-- Create some bean instances from the input source... -->
  <jb:bean beanId="orderItem"  ... ">
    <!-- etc... See Smooks Java Binding docs -->
  </jb:bean>

  <!-- Route bean to camel endpoints... -->
  <camel:route beanid="orderItem">
    <camel:to endpoint="direct:slow" if="orderItem.priority == 'Normal'" />
    <camel:to endpoint="direct:express" if="orderItem.priority == 'High'" />
  </camel:route>

</smooks-resource-list>

In the above example, we route Java Beans from the Smooks BeanContext to the Camel Endpoints. Note that you can also apply templates (e.g. FreeMarker) to these same beans and route the templating result instead of beans (e.g. as XML, CSV or other).

The above configuration shows routing using the ‘beanId’ attribute. It is also possible to route using an attribute named ‘routeOnElement’ For more information about Smooks routing please consult the Smooks User Guide.

The SmooksComponent delegates most of its work to SmooksProcessor which will be described next.

2. SmooksProcessor
Using Smooks Processor gives you full control over Smooks, for example if you want to programatically create the underlying Smooks instance you’d use the SmooksProcessor. Below is an example of using the SmooksProcessor in a Camel route:

Smooks smooks = new Smooks("edi-to-xml-smooks-config.xml");
...
SmooksProcessor processor = new SmooksProcessor(smooks, context);
from("file://input?noop=true").process(processor).
.to("mock:result");

3. SmooksDataFormat
SmooksDataFormat is a Camel DataFormat which is capable of transforming from one dataformat to another and back again. You would use this when you are only interested in transforming from one format to another and not interested in other Smooks features.

Below is an example of using SmooksDataFormat to transform a comma separated value string into a java.util.List of Customer object instances:

SmooksDataFormat sdf = new SmooksDataFormat("csv-smooks-unmarshal-config.xml");
from("direct:unmarshal")
.unmarshal(sdf).convertBodyTo(List.class)
.to("mock:result");

4. Examples

There are currently two very basic examples that demonstrate the usage of Smooks-Camel integration:
1. camel-dataformat (see note below)
2. camel-integration (see note below)

Note on Examples: You must checkout and build all examples before running the Camel examples.

Tom is working on a really interesting example of using UN-EDIFACT in combination with Camel and will be added shortly.

Thanks,

/Daniel