A Step by Step Guide to Replication

Overview

This page is based on the attempt to make sense of ADempiere Replication module of the author. It will be illustrated as a step by step guide for the reader to have the FEEL of how to setup a simple Replication case between a SOURCE and the TARGET machine.

Note: This guide is based on Adempeire 360LTS and GardenWorld

Message Queue

ADempiere use Message Queue (JMS – Java Message Service Protocol) to integrate between systems. Using message queue has many advantages comparing to direct integration at database layer or WebService call. In short, new data from SOURCE machine will be sent to the Message Queue Server in XML format and reside there waiting for the TARGET machine to pick up. This is a more loosely couple way of integration that can guarantee data delivery regardless of the TARGET machine status during the time of delivery. And by using Message Queue Server and XML as medium, integration between different system become more simple.

Mode of Delivery

To be a little bit more specific, JMS support 2 main modes of delivery

  1. Queue: This mode is good for point to point, where there is only 1 target machine. SOURCE send message to a specified queue. TARGET receive message from that queue (and dequeue message), whenever it become available. In other word, TARGET machine can be shutdown during the time of message delivery and will receive message again once it is back online.
  2. Topic: This mode is good for multi cast, where there are multiple target machine receiving the same message. Think about when we subscribe to a magazine (topic). In this approach, TARGET machines must first subscribe to a Topic. Once SOURCE send a message to that Topic, it will be boardcast to all the TARGETs. Note that, the TARGET machine must be Listening during the message sending, otherwise it miss that message.

Note: More on message queue topologies here. The example in this example is based on Topic (Publish and Subscribe) delivery mode.

Message Protocol

ActiveMQ support a 2 well know protocol out of the box.

  1. Openwire: Or TCP protocol. It is fast, flexible but at the same time more complicated. As you shall see, this is the default protocol for ADempiere.
  2. Stomp: A simple, text based protocol. As it is simple, stomp support more platform, including scripting language like Ruby, Perl, Python or PHP. It is also possible that message delivered by TCP will be received by Stomp.
We won’t discuss them in detail here. But before we move on, it is good idea to gain more understanding about ActiveMQ
Image:Note.gif Note:While there are many implementation of JMS Message Queue Server out there in the market, ADempeire is adoption Apache ActiveMQ. Apache ActiveMQ is the most popular and powerful open source messaging and Integration Patterns provider. It is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License

Step by Step Replication Setup in ADempiere

In this simple example, we want to replicate Product Data from SOURCE to TARGET through an ActiveMQ broker. There are 2 side of setting up, which are quite saperated in the context. 1) The setup for SOURCE machine to publish product data to ActiveMQ broker, when change is made. And 2) The setup for TARGET’s Application Server to activate its importing process in order to receive incoming message and save back to its Product table.

As simple as that.

Step 1: Prepare Environment

  1. Prepare 2 ADempiere 360LTS Environment, we will call it SOURCE and TARGET (we won’t discuss how to install Adempeire here).
  2. Install ActiveMQ Server (See Getting Start Guide)
    • Download Binary Distribution, apache-activemq-5.5.0-bin.zip from [1]
    • Unzip and place the folder to C:/apache-activemq-5.5.0-bin
    • That’s all for installation, now start ActiveMQ server by running C:/apache-activemq-5.5.0-bin/bin/activemq.bat
    • ActiveMQ Server is now ready to accept and distribute message!

replicate_1

Image:Note.gif Note:From the console, you will see that ActiveMQ

  • Listening for connections at: tcp://localhost:61616 (tcp / openwire connector).
  • Admin Console is at http://localhost:8161/admin. Here, we can view / manage, connection, queue, topic and etc.
  • Web demo is at http://localhost:8161/demo. A good place to start off understanding ActiveMQ. And also, please check out examples code at C:/apache-activemq-5.5.0-bin/example

Login to ADempiere as GardenWorld admin, open Replication Module, and you will see windows for Export and Import Settings as following picture.

replicate_2

Step 2: Setup Export Processor (on SOURCE machine)

As mentioned earlier, the message will be sent in XML format (i.e., when a new product item is created). Beside writing your own code to generate XML file of product data and put it on the queue, ADempeire Replication Module provides windows / process to help you do just that.

Configurations for Export
  • Replication Strategy: Define what (table/document) and how data (boardcast/reference/local/merge) will be exported by which Export Processor.
  • Export Processor: Define where (i.e., host/port) and how (connection parameters) ADempiere will connect to the ActiveMQ server using which Export Process Type.
  • Export Processor Type: Define the java class that do the real connection and message delivery to ActiveMQ server. This is where the action take place. Out of the package, ADempeire only provide the connection class for Topic delivery mode, org.adempiere.process.rpl.exp.TopicExportProcessor. (once you understand more about how ActiveMQ works, you can write your own class).
  • Export Format: Define what data columns of the tables (from Replication Strategy window) will be exported. ADempeire use information here to generate XML file for export. Note that, the same Export Format must be defined in TARGET as well, ADempiere will use the same Export Format to convert from XML data back to raw data and save to the system.
  • Client’s Model Validator: Set the Export Model Validator and Replication Strategy. This is required only for SOURCE machine, as to trigger Export Process when change is made to the specified data table.

Step 2.1 Define Replication Strategy

A SOURCE can have only one working Replication Strategy (to be defined in Client window), but can have multiple table and document to replicate. In this sample, we want to export only one data, Product.

  • Open Replication Strategy window
  • Create new Replication Strategy record
    • Search Key: MyReplicationStrategy
    • Name: My Replication Strategy
    • Entity Type: User maintained
    • Export Processor: Human Readable name for – JMS Topic Export Processor
  • Create new Replication Table record
    • Replication Strategy: My Replication Strategy
    • Table: M_Product
    • Replication Type: Merge (use merge as it seems to work most here)
    • Entity Type: User maintained
Image:Note.gif Note:Replication Type define the behavior of exporting data (**this is from what I understand from the code, not guarantee on what developer intend**)

  • Broadcast: Repeating sending message to ActiveMQ
  • Local: Do not export, just keep it at SOURCE
  • Merge: Do full merge, create / update / delete
  • Reference: Only update TARGET if data exists at

Step 2.2 Define Export Processor

From previous step, we already define the Export Processor as Human Readable name for – JMS Topic Export Processor. Let’s go through its configuration.

  • Open Export Processor window
  • On Export Process tab, select record as Human Readable name for – JMS Topic Export Processor
    • Search Key: JMS Topic Export Processor
    • Name: Human Readable name for – JMS Topic Export Processor
    • Export Processor Type: Human Readable name for – JMS Topic Export Processor Type
    • Host: localhost (host machine for ActiveMQ)
    • Port: 61616 (default port for tcp)
    • Account: system
    • Password Info: manager

Note: We do not touch on security of ActiveMQ yet, let’s use the default Account/Password for ActiveMQ Server now. More information please see ActiveMQ Security.

  • On Export Processor Parameter Tab, define parameters as following,
    • topicName: MyTopic (this can be any specific topic name)
    • clientID: SourceClient (this can be any client name to be used when connect to ActiveMQ)
    • protocol: tcp
    • timeToLive: 10000 (time before message expire in milliseconds)
    • isDeliveryModePersistent: true (default)

Step 2.3 Define Export Processor Type

From previous step, we already define the Export Processor as Human Readable name for – JMS Topic Export Processor Type. Let’s go through its configuration.

  • Open Export Processor Type window
  • On Export Process Type tab, select record as Human Readable name for – JMS Topic Export Processor Type
    • Java Class: org.adempiere.process.rpl.exp.TopicExportProcessor
Image:Note.gif Note:

  • Out of the package only Topic delivery mode is provided, if you want to use Queue delivery mode, you need to create your own Java Class.
  • HDD Export Process Type (org.adempiere.process.rpl.exp.HDDExportProcessor) is nothing to do with ActiveMQ, it simply export XML into a file server. It is another method of integration without using Message Queue, and as such, we won’t discuss here.

Step 2.4 Define Export Format

Export Format is a crucial part for the replication to work. It tell ADempiere what columns in the specified table will be used to create XML. It use to export as well as to import. As such, the Export Format on SOURCE and TARGET machine must be equal. In this case, we want to export M_Product, which is not listed here yet.

You can create Export Format manually or use the Export Format Generator process.

  • On the SOURCE machine, run Export Format Generator process
    • Window: Product
    • Include only the Tabs that insert records: Checked
    • Include only mandatory columns: UnChecked

Note: During the time of writing, I found a bug which I track it here [2]. Without applying the patch to your system, you can’t run this process. You may test with other table, i.e., C_Bpartner.

  • On the TARGET machine, we will do the same, run Export Format Generator process for Product Export Formats.
  • On both machine, open Export Format window and select M_Product record.
  • Please note that by default only Name and Value is active. In real life, it depend on what column you want to replicate. For our case, select all columns.

replicate_3

 

Image:Note.gif Note:Format Line’s Type

  • XML Element: Normal element, store raw data.
  • Referenced EXP Format: ID column will be referenced with the Name field of its reference table.
  • Embedded EXP Format: Use for Document with Document Lines, ie., C_Order have a column Order Line which reference to C_OrderLine Export Format

Step 2.5 Define Client’s Model Validator

  • Open Client window
    • Add Export Model Validation Classes: org.adempiere.model.ExportModelValidator
    • Set Replication Strategy: My Replication Strategy
That’s all for setup on SOURCE machine.

Step 3: Setup Import Processor (on TARGET machine)

Step 3.1 Define Export Format

We want to make sure that the SOURCE and TARGET machine have the same set of Export Format, in this case for Product data. If not done yet, repeat step 2.4, this time for TARGET machine.

Step 3.2 Define Import Processor

Import Processor is the process that will be run as scheduled job by Application Server. Its duty is to Listen to the specified topic on ActiveMQ server.

  • Open Import Processor window
  • On Import Process tab, select record – Human Readable name for – JMS Topic Import Processor
    • Activate this record.
    • Frequency: 1 Minutes (for testing purposes)
    • Host: localhost (host machine for ActiveMQ)
    • Port: 61616 (default port for tcp)
    • Account: system
    • Password Info: manager
  • On Import Processor Parameter Tab, define parameters as following,
    • topicName: MyTopic (same as Export Processor)
    • clientID: TargetClient
    • protocol: tcp
    • isDurableSubscription: true
    • subscriptionName: MySubscription

Step 3.3 Define Import Processor Type

From previous step, we already define the Export Processor as Human Readable name for – JMS Topic Import Processor Type. Let’s go through its configuration.

  • Open Import Processor Type window
  • On Export Process Type tab, select record as Human Readable name for – JMS Topic Export Processor Type
    • Java Class: org.adempiere.server.rpl.imp.TopicImportProcessor
Image:Note.gif Note:

  • Out of the package only Topic delivery mode is provided, if you want to use Queue delivery mode, you need to create your own Java Class.
  • HDD Import Process Type (org.adempiere.server.rpl.imp.FileImportProcessor) is nothing to do with ActiveMQ, it simply get XML from file server. It is another method of integration without using Message Queue, and as such, we won’t discuss here.

Step 4: Validate Export & Import XML File

One of the frequent problem that the replication is not through is about incorrect formatting. ADempeire provide a process to help us validate that the SOURCE and TARGET machine have compatible format.

  • On SOURCE machine, Run Process Test Export Model.
    • Export Format: Product
    • File: D:\Product_Format.xml
    • Click Start to test
  • On SOURCE machine, Run Process Test Import Model.
    • Export Format: Product
    • File: D:\Product_Format.xml
    • Click Start to test

No error message shown, we are ready to start replication!!!

Step 5: Test Replication

  • Run ActiveMQ server
    • If ActiveMQ server is not started, start it and go to URL http://localhost:8161/admin
    • Click on Topic link, now you will see that there is no topic MyTopic on the list.

replicate_4

  • Start Adempiere Application Server (make sure that it is run for the TARGET machine/database)
    • Open ADempiere Web Admin http://localhost/admin and click on ADempiere Server Management
    • Now, you should see the new Import Process running

replicate_5

Now we can start testing replication
  • On SOURCE machine, create new Product record.
    • Search Key: MyTestProduct
    • Name: My Test Product
    • UOM: Kilogram

Note: The default UOM Each is not usable as it is created by System. For the Product data to be replicated, all reference field data must be client data (GardenWorld)

  • Click Save to create new record, the Product data is now published to the ActiveMQ as you can see MyTopic is now listed in topic page.

replicate_6

  • Wait for a while, and the Product should be replicated to TARGET machine. You can open and review the new Product item in TARGET machine.
  • If there is any error, you can see from ADempiere’s Application Server Console.
  • To see the log on TARGET machine, open Import Processor | Log window

Remarks / Issues / Concern

All in all, from my experiment, the module is working as it should. But it is not false tollerance, everything have to be set correctlly for it to function. Looking through source code is a required steps to understand it.

Problem – Message not get delivered
  • Make sure that the Application Server is Listening to the Topic, before the message is sent. Remember, this is Topic (Publish and Subscribe) mode.
  • Data is very sensitive. Make sure that all the required fields are exported, otherwise, it can create Null Error when create new record at TARGET.
  • All data must be as of the same client, i.e., Product’s UOM form the SOURCE is using System’s UOM, i.e., “Each”, when reach TARGET, “Each” is not allowed, as it is no Client’s data. Make sure that any reference data is of the same client.
  • Whenever you change the Export Format at TARGET, always restart the Import Process Job.
  • (Seem to me that) Both Name and Value are used as reference when things get updated. As such, both field can not change, otherwise, it can’t find the TARGET record for update.
How to debug Server’s Import Process
  • Add main function in org.compiere.server.ReplicationProcessor.java and run it from Eclipse
    • Running it from eclipse equal to run by Application Server, as such it will be listening to the event.
    • Note that 1000000 below is the Process ID. This ID will be replaced by the ID of the Import Process you want to debug.
	/***************************************************************************
	 * 	Test
	 *	@param args ignored
	 */
	public static void main (String[] args)
	{
		Adempiere.startup(true);
		MIMPProcessor model = new MIMPProcessor (Env.getCtx(), 1000000, "1001");
		ReplicationProcessor rp = new ReplicationProcessor(model);
		rp.start();
	}
Miscellaneous Findings
  • How Export Format is selected to use when there is more than 1 Export Format for a table?
    • Created before –> Active Status –> Later version of Export Format will be used???
  • What happen with the message that got error during import on TARGET machine? How to re-import?

See Also