Archive for 'Camel'

This post is about the work we have done to improve Smooks OSGI integration.

With Smooks 1.4 we added support for OSGI in the form of making the smooks-all.jar into a bundle. This jar contains all the Smooks classes and resources and is a convenience jar that can be used if you don’t want to keep track of all the individual jars that make up Smooks. During this work we also added a few classes to make Smooks work well inside an OSGI container.

For some background and more details about the Smooks-OSGI work please take a look at this wiki page.

Using Smooks in an OSGI Container (with Camel and ServiceMix)
With 1.4 we added Camel support which has had OSGI support for a long time. We added a few basic examples but it turned out that this was not the best solution. The solution in 1.4 required users to explicitly specify the packages the client bundle required and this made maintenance difficult.
Using Smooks in Camel should not require anything more then declaring that the client bundle uses Camel and perhaps Spring. Client bundles now only need to declare that they import packages for Camel, for example:

Manifest-Version: 1.0
Export-Package: example;uses:="org.apache.camel,
    org.springframework.context.support,
    org.springframework.context"
Bundle-Version: 1.0
Bundle-Name: Milyn Smooks Example - Smooks Camel CVS to XML
Bundle-ManifestVersion: 2
Import-Package:
    example,org.apache.camel;version="2.6",
    org.springframework.context;version="1.2.0",
    org.springframework.context.support;version="1.2.0"
Bundle-SymbolicName: milyn-smooks-example-camel-csv-to-xml

Simplified deployment to ServiceMix
ServiceMix users will hopefully be glad to know that we now generate a features.xml file for easy deployment into service mix. A features.xml specifies all the bundles that a feature need and can be installed/uninstalled as a single unit.
For example, to install the Smooks bundle you can use the following commands:

karaf@root> features:addUrl mvn:org.milyn/milyn-smooks-all/1.5-SNAPSHOT/xml/features
karaf@root> features:install smooks

UN/EDIFACT
Smooks has UN/EDIFACT support which includes pre-generated artifacts that are available in maven. As part of the OSGi work, we have modified these artifacts so as to make them OSGi fragment bundles

Users can now install the mappings and binding as fragment bundles and these will attach to a specific version of the Smooks bundle. This is what this looks like running in Apache ServiceMix 4.3.0.1-fuse:

[ 430] [Active ] [ ] [ ] [ 60] Smooks OSGi (1.5.0.SNAPSHOT)
Fragments: 439,441
[ 439] [Resolved ] [ ] [ ] [ 60] Smooks EDI - UN/EDIFACT - D96A - Bindings (1.4)
Hosts: 430
[ 441] [Resolved ] [ ] [ ] [ 60] Smooks EDI - UN/EDIFACT - D96A - Mapping Model (1.4)
Hosts: 430
[ 448] [Active ] [ ] [Started] [ 60] Milyn Smooks Example - Smooks Camel UNEDIFACT to String (1.0)

Above we can see that the binding (id=439) and the mapping (id=441) have been attached to the Smooks bundle (id=420). More fragment bundles can be attached as required and you can attach the same fragment to different version of the Smooks bundle since we specify a version on the Fragment-Host header.

As mentioned earlier in this post you can find more information and details about this work on this wiki page.

Two of the main new features in Smooks v1.4 are, out of the box support for UN/EDIFACT Interchanges, and Apache Camel integration.

The new UN/EDIFACT support allows you to process UN/EDIFACT Interchanges by either or both of the following methods:

  1. Conversion to XML, allowing you to use standard XML tools (or other Smooks extensions) for further processing.
  2. Binding the interchange message data into pre-built Java object models, or custom models if you prefer to use your own.

All the required artifacts for reading any of the UN/EDIFACT directories (and converting to XML) are available in Maven, as are all of the pre-built Java Object models for all the messages defined in all of the UN/EDIFACT directories.  To use them, all you need to do is include the appropriate artifacts into your project.  See the User Guide.

The new Apache Camel integration allows you to hook Smooks into your Camel routes as a Camel Component or Camel DataFormat.  Another important part of the Camel integration are the Smooks extensions that allow you to perform inline routing of Smooks BeanContext data to Camel endpoints i.e. you can also process huge (GB+) data streams.

Putting these two new capabilities together allows us to provide some interesting functionality with respect to Splitting and Routing UN/EDIFACT message Interchanges.  We use Smooks to perform the reading, conversion (to XML and/or Java) and splitting of the interchanges and then use Camel to to perform the inline routing of the Java or XML (using the new Camel extensions for Smooks).

We created an example that demonstrates this capability:

Smooks-Camel integration revisited

This entry is about the work we have done integrating Smooks with Apache Camel for Smooks v1.4.  With this tighter integration of these frameworks, you can leverage the power of Smooks to process/transform a wide range of data types, with the power of Apache Camel to route data to a huge range of endpoint types.

Tom started this work a long time ago and got feedback from Claus Ibsen, which Tom implemented. Later Christian Mueller improved the code and I’ve continued working on that code base.

When it comes to using Smooks in Camel there are three options of usage:

1. SmooksComponent
The SmooksComponent is a Camel component which can be used in a Camel route like this:

from("file://inputDir?noop=true")
.to("smooks://smooks-config.xml")
.to("jms:queue:order")

This route will read files from the ‘inputDir’ directory and pass the contents of those files to the Smooks Component. The file will be automatically converted into an instance of javax.xml.transform.stream.StreamSource. TypeConversion is a built-in Camel feature that you can read more about here.

So we know what the source type is but how about the Result type we want Smooks to produce? If you have been following the latest blog posts you might have already guessed that the type(s) are configured in ’smooks-config.xml’ as described in this blog post.

Using the SmooksComponent you can use all the features from Smooks including routing to Camel endpoints. This enables fragments to be routed during transformation. This means that the routing can be performed as the message is being filtered Vs as a batch operation after filtering the message, making it easier to handle large/huge message streams.

Here is an example of routing to camel endpoints using Smooks:

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
         xmlns:camel="http://www.milyn.org/xsd/smooks/camel-1.4.xsd">

  <!-- Create some bean instances from the input source... -->
  <jb:bean beanId="orderItem"  ... ">
    <!-- etc... See Smooks Java Binding docs -->
  </jb:bean>

  <!-- Route bean to camel endpoints... -->
  <camel:route beanid="orderItem">
    <camel:to endpoint="direct:slow" if="orderItem.priority == 'Normal'" />
    <camel:to endpoint="direct:express" if="orderItem.priority == 'High'" />
  </camel:route>

</smooks-resource-list>

In the above example, we route Java Beans from the Smooks BeanContext to the Camel Endpoints. Note that you can also apply templates (e.g. FreeMarker) to these same beans and route the templating result instead of beans (e.g. as XML, CSV or other).

The above configuration shows routing using the ‘beanId’ attribute. It is also possible to route using an attribute named ‘routeOnElement’ For more information about Smooks routing please consult the Smooks User Guide.

The SmooksComponent delegates most of its work to SmooksProcessor which will be described next.

2. SmooksProcessor
Using Smooks Processor gives you full control over Smooks, for example if you want to programatically create the underlying Smooks instance you’d use the SmooksProcessor. Below is an example of using the SmooksProcessor in a Camel route:

Smooks smooks = new Smooks("edi-to-xml-smooks-config.xml");
...
SmooksProcessor processor = new SmooksProcessor(smooks, context);
from("file://input?noop=true").process(processor).
.to("mock:result");

3. SmooksDataFormat
SmooksDataFormat is a Camel DataFormat which is capable of transforming from one dataformat to another and back again. You would use this when you are only interested in transforming from one format to another and not interested in other Smooks features.

Below is an example of using SmooksDataFormat to transform a comma separated value string into a java.util.List of Customer object instances:

SmooksDataFormat sdf = new SmooksDataFormat("csv-smooks-unmarshal-config.xml");
from("direct:unmarshal")
.unmarshal(sdf).convertBodyTo(List.class)
.to("mock:result");

4. Examples

There are currently two very basic examples that demonstrate the usage of Smooks-Camel integration:
1. camel-dataformat (see note below)
2. camel-integration (see note below)

Note on Examples: You must checkout and build all examples before running the Camel examples.

Tom is working on a really interesting example of using UN-EDIFACT in combination with Camel and will be added shortly.

Thanks,

/Daniel

Apache Camel + Smooks

I did a little playing with Camel over the weekend… had a go at integrating Smooks.

I think those using Camel may find it interesting as another option.  The integration was simple enough and can be seen in this svn workspace (including tests etc).  With better knowledge of Camel, I’m sure there’s a better way of integrating it, but at least it’s a start.

At the moment, you can map String or Java results into the out-message of the Camel Exchange, so you can do templating or java binding (not exposing validation results yet) of XML and non-XML data.  So you can do lots of the normal Smooks type processing.

The aspect I think most Camel users might find interesting is the potential wrt splitting and routing of huge messages (XML and non-XML).  Basically… being able to split and route huge messages as you stream them i.e. not the two step method that seems to be there at the moment – 1) split out all messages 2) route all messages.

Lets look at a simple example, where we have an XML message stream containing customer account activity logs (or… could be CSV formatted).  We want to route each log event to JMS queues based on nationality.

The RouteBuilder config (can also configure the SmooksProcessor via Java ala DSL) …

public class MyRouteBuilder extends RouteBuilder {

    public void configure() throws Exception {

        from("file:src/data/logs?noop=true").
            process(new SmooksProcessor("/configs/log-split.xml", this));

        from("jms:queue:ireland").process(new LogProcessor("ie"));
        from("jms:queue:greatbritain").process(new LogProcessor("gb"));
    }
}

And then the Smooks “log-split.xml” config…

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
          xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.3.xsd"
          xmlns:camel="http://www.milyn.org/xsd/smooks/camel-1.4.xsd">

    <!-- Create a new ActivityLogEvent for each log entry... -->
    <jb:bean beanId="logEvent" class="com.acme.activity.ActivityLogEvent"
                                  createOnElement="activity/event">
        <jb:value property="transactionId" data="event/@txId" />
        <jb:value property="customerId" data="event/customer/@customerId" />
        <jb:value property="nationality" data="event/customer/countryCode" />
        <!-- etc... -->
    </jb:bean>

    <!-- Route GB based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:greatbritain"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'GB'</condition>
    </camel:router>

    <!-- Route Irish based customer activity... -->
    <camel:router beanId="logEvent" toEndpoint="jms:queue:ireland"
                                 routeOnElement="activity/event">
        <condition>logEvent.nationality == 'IE'</condition>
    </camel:router>

</smooks-resource-list>

The <jb:bean> config creates an ActivityLogEvent bean instance for every activity log. The following <camel:router> configs conditionally route the ActivityLogEvent bean instances to the specified JMS endpoints based on the country-code on the event log.  Of course, using Smooks Templating, you could apply a template to the ActivityLogEvent to produce some other data format and route that instead (a simple template generated String bean bound into the Smooks BeanContext).

The size of the event log is irrelevant here – could be GBs.  Only one ActivityLogEvent instance is in memory at any given time (a new instance created for each event).  The routing is performed as the event log is being streamed i.e. not batched up for routing in a later step.  Assumption here is that there’s a way of telling the Camel File component to forward the file as a stream/reader.

The current implementation is available in SVN here.  Since I’m not overly familiar with Camel, I’m sure there are things we could do better/different.  Please let us know!!!