Smooks Data Integration » Smooks

Processing UN/EDIFACT message Interchanges…

Tom Fennelly — Mon, 28 Jun 2010 12:47:30 +0000

Bård (Langöy) and I have been doing lots of work on improving our EDI support, specifically in the area of handling UN/EDIFACT (WikipediA) message Interchanges. The following improvements are now in the Smooks v1.4 codebase (available from the 1.4-SNAPSHOT):

A new UN/EDIFACT Interchange Reader (), which allows Smooks to process UN/EFIFACT message Interchanges (one or more UN/EDIFACT messages wrapped in the UN/EDIFACT Interchange control segments).
A new Maven/Ant tool called the “EDI Conversion Tool” (ECT) that can take the official UN/EDIFACT message definition directory zip files and from them, generate a jar file containing a set of equivalent Smooks EDI Mapping Models. This tool is very easy to configure and use, but we are in the processing of pre-generating a library of these jars for all the UN/EDIFACT message sets and making them available trough the central maven repository, making it even easier to consume UN/EDIFACT messages with Smooks. We hope to add support for other formats (X.12, HL7 etc) in future releases of Smooks.

So that’s what’s available in the current 1.4 codebase (1.4-SNAPSHOT), but we’re also in the process of making more additions to this for Smooks v1.4. We’re extending the EDI Java Compiler (EJC) tools to support Java Model generation from the EDI Mapping Model sets generated by the ECT tool (mentioned above). We’ll also make these available as pre-built jar files from the Maven repository.

Another improvement we’ve made in the 1.4 codebase is the addition of support for Java Model to EDI serialization on the EJC generated Java object models, meaning we’ll be able to do full round trip binding of an EDI (and UN/EDIFACT) message to a populated Java Object model that can be modified and then serialize back out to an EDI stream. Or, your app can “manually” construct the same Java Object models and serialize them out to an EDI Stream (think JAXB).

Generating a Mapping Model Set using ECT

The easiest way to consume UN/EDIFACT messages using Smooks is to use ECT (EDI Conversion Tool) to generate the EDI Mapping Model set (if we haven’t already pre-built them). The steps are really easy:

Download the official UN/EDIFACT message definitions directory zip file you are interested in from the UNECE website.
Create a maven module for the EDI Mapping Model set to be generated and add the maven-ect-plugin (see below) to it’s POM.
Execute “mvn clean install” in your maven module and the EDI Mapping Model set jar file will be generated as normal in the modules target folder, and installed into your local maven repository.

The following is an example of the maven module POM for generating the EDI Mapping Model set jar for the d03b.zip definitions directory zip file.



    4.0.0

    org.milyn.edi.unedifact
    d03b-mapping
    1.0.SNAPSHOT
    UN/EDIFACT - D03B - Mapping Model

    
        
            
                org.milyn
                maven-ect-plugin
		        1.4-SNAPSHOT
                
                    d03b.zip
                    UNEDIFACT
                
                
                    
                        generate

Of course when we execute this step, we will perform “mvn clean deploy” and install the jar file in the public maven repository, making it publicly available and thereby removing the need for you to perform this step. A peek into the public SNAPSHOT repo shows that we’ve started this process.

Using a Mapping Model Set on the UN/EDIFACT Reader

Using a generated EDI Mapping Model set jar in your application (via Smooks) is trivial. You need to:

Add the generated EDI Mapping Model set jar to your classpath (e.g. using a maven dependency configuration).
Add the configuration to your Smooks configuration, using the generated jar’s maven groupId, artifactId and version to reference that particular mapping model set through a URN (see below).

An example of the configuration for the above generated EDI Mapping Model set jar for the d03b.zip definitions directory zip file would be as follows:

Using the “urn:::” URN pattern, Smooks can find the EDI Mapping Model set jar file on the classpath,

And that’s about it. The associated Smooks instance will accept a UN/EDIFACT message interchange containing one or more messages defined in the d03b.zip definitions directory zip file, and will generate a stream of SAX events into Smooks, which can be processed using all the other Smooks capabilities (Java Binding, Validation, Templating etc). As stated earlier, the improvements we are working on for EJC will result in us being able to generate Java Object models from an EDI Mapping Model set jar.

Play with the Examples… and please give feedback

Check out and build the Smooks examples from https://svn.codehaus.org/milyn/trunk/smooks-examples (mvn install).

Change into the “ect-unedifact” directory after building all the examples (must build them all so as to install the poms) and run the example using mvn exec:java. The ect-unedifact example depends on the “d03b-mapping” module which has been pre-built and is available in the public SNAPSHOT repo.

And guys…. PLEASE let us know how you get on with this stuff… things you like and dislike etc !!!

posted in EDI by Tom Fennelly Leave A Comment

Introduction: Using a factory to instantiate beans with the javabean cartridge

Maurice — Mon, 12 Oct 2009 08:01:59 +0000

I would like to introduce the new factory feature in the Javabean cartridge of the up coming Smooks 1.3 release. This new features makes it possible to use a static factory method or a factory object to instantiate objects with the Javabean cartridge.

Before this new feature you could only instantiate new beans by providing the class name of the bean and the bean would then be created using it’s public parameterless constructor. So it was mandatory to have a parameterless constructor on the Javabean. Here is an example:

But now it is possible to use a factory to instantiate the bean. So you don’t need a public parameterless constructur anymore. You even don’t have to define the actual class name any more. Any of the interfaces that the class implements suffices. However only the methods of that interface are available for binding then. Here is an example where we instantiate an ArrayList object using a static factory method:

With the factory attribute you set the factory definition. The class attributes defines that a List object is returned. What kind of List object (ArrayList, LinkedList) is up to the ListFactory to decide. Here is another example:

Here we define that an instance of the ListFactory needs to be retrieved using the static method getInstance() and that then the newList() method needs to be called on the ListFactory object to create the List object. This construct makes it possible to easily call Singleton Factories.

The default factory definition language looks like this: factory.class#static_method{.instance_method}

It is also possible to use MVEL as the factory definition language. MVEL has some advantages over the basic default definition language, for example you can use objects from the bean context as the factory object or you can call factory methods with parameters. These parameters can be defined within the definition or they can be objects from the bean context. To be able to use MVEL you must set the ‘factory.definition.parser.class’ global parameter to ‘org.milyn.javabean.factory.MVELFactoryDefinitionParser’.

Here is an example using MVEL:



	
		
			org.milyn.javabean.factory.MVELFactoryDefinitionParser

Maybe you wonder why we don’t use MVEL as the default factory definition language? Currently the performance of the basic definition language and MVEL is about equal. The reason that the basic definition language isn’t faster is because it currently uses reflection to call the factory methods. However there are plans to use byte code generation instead of reflection. This should improve the performance dramatically. If MVEL where the default language then we couldn’t do anything to improve the performance for those people who don’t need any thing more then the basic features that the basic definition language offers.

If you have any questions or remarks then I would gladly hear them.

posted in Java Binding by Maurice Leave A Comment

Data Validation with Smooks v1.2

Daniel Bevenius — Thu, 21 May 2009 13:43:18 +0000

Smooks 1.2 adds support for message data Validation as one of its new features. This new feature allows you to perform strong field and fragment validation on not just XML data, but also on EDI, JSON, CSV etc. It currently supports Regex and MVEL rules bases (Drools to follow). Regex rules allow you to perform low level field format validation, while MVEL rules allow you to perform Business rules validation at a fragment/message level.

This is not intended as an alternative to Schema based validation such as XSD or Relax. It provides additional layers of validation in areas not covered by these mechanisms.

Examples of validations one might perform on an Order message containing a collection of Order Item:

1. Use a Regex rule to validate that an orderId field follows a predefined format of “Uppercase Char, followed by 5 digits” e.g. “W12345″ or “U54321″.

2. Use an MVEL rule to enforce a business rule by validating that an order item total is not greater than 50.00 i.e. “price * quantity < 50.00″. Of course these can be more complex.

The easiest way to get familiar with the validation features is to look at an example, so here goes.

For the following example assume that we have the xml that looks like this:


    
        A188127
        user1
        
            Harry
            Fletcher
        
        harry.fletcher@gmail.
        South Dakota

An example of a validation configuration could look like this:

The ‘exeuteOn‘ attribte specifies the fragment or attribute that you want the validation rule to be applied to. In the case of a Regex rule, it specifies a specific field of data (attribute or element text), while in the case of an MVEL rule, it simply identifies the fragment that triggers execution of that rule.

The next thing to notice is the ‘name‘ attribute. This attribute refers to a rule in one of the configured ruleBase. In the case of the example above, we refer to the ‘email‘ rule in the ‘customer‘ ruleBase i.e. ‘customer.email‘. You can define as many ruleBase elements and validation element as you need, but we are only using one of each here to keep this easy to read.

The ‘customer’ ruleBase is handled by the RegexProvider, as defined in the ‘provider’ attribute on the ruleBase configuration. We can also see that the ‘customer’ ruleBase rules are located in a properties file named ‘rules/customer.properties‘:

# Email address...
email=^[\\w-\\.]+@([\\w-]+\\.)+[\\w-]{2,4}$

Next, lets take a look at how we execute Smooks to capture some validation results in a ValidationResult instance:

final Smooks smooks = new Smooks("smooks-config.xml");
try
{
    // Create an exec context - no profiles....
    final ExecutionContext executionContext = smooks.createExecutionContext();
    final ValidationResult validationResult = new ValidationResult();

    // Filter the input message...
    smooks.filterSource(executionContext, new StringSource(messageIn), validationResult);

    for (OnFailResult result : results.getWarnings())
    {
        System.out.println("\t" + result.getMessage());
    }
}
finally
{
    smooks.close();
}

As you can see, Validation faiures can be reported at different levels of severity. This is controlled by setting ‘onFail‘ to one of ‘OK‘, ‘WARN‘, ‘ERROR‘, or ‘FATAL‘. Setting onFail to ‘FATAL’ will cause an exception to be thrown and processing to halt. The other three levels will not generate an exception. Failures at these lower severity levels can be accessed through the ValidationResult instance supplied to the Smooks.filterSource method. An example of getting WARN level failures is shown above with the result.getWarnings() call.

Having usable validation reports is important and Smooks validation supports localized messages for validation failures. These messages can be defined in standard Java ResourceBundle files (.properties format). A convention is used here, based on the rule source name (”src“). The validation message bundle base name is derived from the rule source (”src“) by dropping the rule source file extension and appending a “_messages“.

So for our example we could have a file named ‘customer_messages.properties‘ containing our default locale results:

email=ftl:Invalid email address '${ruleResult.text}' at '${path}'.

So calling result.getMessage() for a validation failure on ‘customer.email’ rule will result in the following message:

Invalid email address ‘harry.fletcher@gmail.’ at ‘/Order/header/email’.

And we could also add a message bundle file for Swedish users. Following the ResourceBundle naming convention, it should be named ‘customer_messages_sv_SE.properties‘ and would contain the Sweish localized messages:

email=ftl:Felaktig epost adress '${ruleResult.text}' i '${path}'.

Calling result.getMessage(new Locale(”sv, “SE”)) will produce:

Felaktig epost adress ‘harry.fletcher@gmail.’ i ‘/Order/header/email’.

We have only shown a portion of the validation features here. More information can be found in the Smooks User Guide. An example of the validation features can be found in Validation Example.

posted in Blog by Daniel Bevenius Leave A Comment

Smooks Persistence part 2: How to use Hibernate/JPA

Maurice — Wed, 25 Mar 2009 11:20:09 +0000

With the new Smooks Persistence cartridge in Smooks 1.2 you can directly use several entity persistence frameworks from within Smooks. In this post I will show you how this works with Hibernate and any other JPA compatible framework.

This post is part of a multi post article. You can find the previous post here: Smooks Persistence part 1: The Introduction

In this post we will look at an example of how to persist Hibernate Entities. This example is based on the example of the previous post.

First, lets take a look at the data we are going to process. In this example we are processing XML data. It should be noted however, that the input data could also be CSV, JSON, EDI, Java or any other structured/hierarchical data format. The same principals apply, no matter what the data format is!

These are the Hibernate entities we are going to use:

@Entity
@Table(name="orders")
public class Order {

	@Id
	private Integer ordernumber;

	@Basic
	private String customerId;

	@OneToMany(mappedBy = "order", cascade = CascadeType.ALL)
	private List orderItems = new ArrayList();

	public void addOrderLine(OrderLine orderLine) {
		orderItems.add(orderLine);
	}

       // Getters and Setters....
}

@Entity
@Table(name="orderlines")
public class OrderLine {

	@Id
	@GeneratedValue(strategy=GenerationType.IDENTITY)
	private Integer id;

	@ManyToOne
	@JoinColumn(name="orderid")
	private Order order;

	@Basic
	private Integer quantity;

	@ManyToOne
	@JoinColumn(name="productid")
	private Product product;

       // Getters and Setters....
}

@Entity
@Table(name = "products")
@NamedQuery(name="product.byId", query="from Product p where p.id = :id")
public class Product {

	@Id
	private Integer id;

	@Basic
	private String name;

       // Getters and Setters....
}

The Smooks configuration looks like this:




	
	

		
		
		
	

	
	

		
		
		
	

	
	

		from Product p where p.id = :id

If we want to use the named query ‘productById’ instead of the query string then the DAO locator configuration will look like this:

The following code executes Smooks. Note that we use a SessionRegister object so that we can access the Hibernate Session from within Smooks.

        // Note: In a real world application you should cache the Smooks instance
        // else you will have enormous performance penalties.
        Smooks smooks = new Smooks("smooks-config.xml");

        ExecutionContext executionContext = smooks.createExecutionContext();

        // The SessionRegister provides the bridge between Hibernate and the
        // Persistence Cartridge. We provide it with the Hibernate session.
        // The Hibernate Session is set as default Session.
        DaoRegister register = new SessionRegister(session);

        // This sets the DAO Register in the executionContext for Smooks
        // to access it.
        PersistenceUtil.setDAORegister(executionContext, register);

        Transaction transaction = session.beginTransaction();

        Source source = new StreamSource(new File("order.xml"));        

        smooks.filter(source, null, executionContext);

        transaction.commit();

If you want to take a JPA version of this example for a spin, simply check out the examples from subversion (http://svn.codehaus.org/milyn/trunk/smooks-examples/) and you’ll find it in the “dao-router” folder. It also shows you how to use custom DOA’s with the Persistence Cartridge.

With Scribe and the Persistence Cartridge it is very easy to use Hibernate, JPA and most other persistence frameworks within Smooks. Later on in this series I will explain how too write your own DAO Adapters, so that you can add support for other persistence frameworks.

In the next post of the series I will explain how you can use Ibatis with the Persistence Cartridge.

posted in Java Binding by Maurice Leave A Comment

Smooks Persistence part 1: The Introduction

Maurice — Thu, 26 Feb 2009 11:31:44 +0000

One of the major new features in Smooks v1.2 will be the new Persistence Cartridge. This cartridge enables the use of several entity persistence frameworks from within Smooks, currently targeting Hibernate, Ibatis and any JPA compatible framework. It also allows you to use your own Data Access Objects (DAOs).

This cartridge is great for those cases where you already have your data access layer and want to use its power from within Smooks. It also allows you to reuse these persistence resources on any format of data, not just XML e.g. EDI, CSV, JSON etc.

This is the first post of a series of posts about the persistence cartridge. This post is an introduction of the cartridge. I will give an overview of what the cartridge can do and also show an example where I use custom DAO’s to persist the data of an XML document. In the next posts in this series, I will show examples of how Hibernate, JPA and Ibatis entities can be used.

The Persistence cartridge is build around a data access layer abstraction library called “Scribe”. Scribe is a new Milyn subproject, born out of the needs of the Persistence cartridge. It enables the Persistence cartridge to use your DAOs without it actually needing to know the details of your DAOs. In fact, they don’t need to be DAOs at all. You can also use any entity persistence framework without the Persistence cartridge seeing the difference. Scribe supports abstracting the default actions you can expect from any DAO or persistence library like inserting, updating, deleting and querying for entities.

For Scribe to understand how to use your DAOs, it needs a mapping. Scribe offers two methods for doing this:

Java Annotations
Scribe has a set of Java Annotations like @Insert, @Update, @Delete and @Locate. These annotations enable you to map your DAOs to the DAO actions.
Java Interfaces
Scribe defines a set of Java interfaces like DAO, Locator and Queryable. You build your DAOs By combining and implementing these interfaces. These interfaces are primarily meant to be used for implementing adapters for persistence frameworks.

The Persistence Cartridge defines a set of Smooks Visitor components that use the DAO actions defined by Scribe. For example, the Inserter visitor can insert your entities by calling the method defined by the DAO action mapping. We see an example of this later.

The cartridge has the following Smooks Visitor components:

Inserter
Inserts/Persists entities
Updater
Updates/Merges entities
Deleter
Deletes entities
Locator
Locates entities with a query or with your own lookup method.

Now that you have an idea of what the Persistence cartridge can do, let’s take a look at a DAO based example. We will see Hibernate, IBatis and JPA examples in the followup posts in this series.

The example will read an XML file containing order information (note that this works just the same for EDI, CSV etc). Using the javabean cartridge, it will bind the XML data into a set of entity beans. Using the id of the products within the order items (the element) it will locate the product entities and bind them to the order entity bean. Finally, the order bean will be persisted.

The order XML message looks like this:

The following custom DAO will be used to persist the Order entity:

@Dao
public class OrderDao {

   private final EntityManager em;

   public OrderDao(EntityManager em) {
     this.em = em;
   }

   @Insert
   public void insertOrder(Order order) {
     em.persist(order);
   }
}

When looking at this class you should notice the @Dao and @Insert annotations. The @Dao annotation declares that the OrderDao is a DAO object. The @Insert annotation declares that the insertOrder method should be used to insert Order entities.

The following custom DAO will be used to lookup the Product entities:

@Dao
public class ProductDao {

   private final EntityManager em;

   public ProductDao(EntityManager em) {
      this.em = em;
   }

   @Lookup(name="id")
   public Product findProductById(@Param("id") int id) {
      return em.find(Product.class, id);
   }
}

When looking at this class, you should notice the @Lookup and @Param annotation. The @Lookup annotation declares that the ProductDao#findByProductId method is used to lookup Product entities. The name parameter in the @Lookup annotation sets the lookup name reference for that method. When the name isn’t declared, the method name will be used. The optional @Param annotation let’s you name the parameters. This creates a better abstraction between Smooks and the DAO. If you don’t declare the @Param annotation the parameters are resolved by there position.

The Smooks configuration look likes this:

The following code executes Smooks:

Smooks smooks = new Smooks("./smooks-configs/smooks-dao-config.xml");
ExecutionContext executionContext = smooks.createExecutionContext();

// The register is used to map the DAO's to a DAO name. The DAO name isbe used in
// the configuration.
// The MapRegister is a simple Map like implementation of the DaoRegister.
DaoRegister register =
	MapRegister.builder()
		.put("product", new ProductDao(em))
		.put("order", new OrderDao(em))
		.build();

PersistenceUtil.setDAORegister(executionContext, mapRegister);

// Transaction management from within Smooks isn't supported yet,
// so we need to do it outside the filter execution
EntityTransaction tx = em.getTransaction();
tx.begin();

smooks.filter(new StreamSource(messageIn), null, executionContext);

tx.commit();

If you want to take this example for a spin, simply check out the examples from subversion (http://svn.codehaus.org/milyn/trunk/smooks-examples/) and you’ll find it in the “dao-router” folder. It also shows you how to use JPA directly, without any DAOs in between.

The cartridge is almost ready for it’s first release. There is one thing that I am still not completely sure about and it would be great if people could provide some feedback. I am not completely happy about the action names: insert, update and delete. In the world of persistence frameworks, different sets of names are used for these actions. JPA, for instance, uses persist, merge and remove. Hibernate (outside of JPA) uses save, update and delete and Ibatis uses insert, update and delete. Which names would you choose and why?

In the next post of the series I will explain how to use Hibernate directly.

posted in Java Binding by Maurice Leave A Comment

A Mule podcast about Smooks and “Smooks for Mule”

Maurice — Mon, 02 Feb 2009 11:01:58 +0000

MuleSource recently posted a Podcast about “Smooks for Mule” on their “From the Mule’s mouth” blog. In the Podcast, I get interviewed by Ross Mason about “Smooks for Mule” and “Smooks”. I talk about why I started Smooks for Mule and what Smooks can do. I outline some examples of where Smooks is in use today, as well as what people can expect in the next releases of “Smooks for Mule” and “Smooks”.

You can listen to the podcast via the player on the end of this post or you can go to the original Mule blog post. I hope that you find it interesting and useful.

I would like to thank MuleSource for enabling this interview.

posted in Podcast by Maurice Leave A Comment

The start of the official Smooks blog

Maurice — Fri, 23 Jan 2009 13:17:57 +0000

Yeah, we of the Smooks project are also going to do it. Start blogging about our project. The blog is the perfect place to inform you about the latest things that are going on. With this blog we want to get our idea’s out there and introduce you to the latest features of Smooks. You can also expect posts with small tutorials featuring common, experimental or less know features of Smooks or one of it’s child projects.

The blog itself is in it’s Beta phase right now. So you can expect this blog to change it’s appearance over the next couple of months.

I hope you will enjoy this blog and find it usefull. Please place comments if you comments or questions. We will try to answer them all.

Regards,

Maurice

posted in Blog by Maurice Leave A Comment