Message flow between MQ and Datastage

Message flow between MQ and Datastage

INTRODUCTION

This document guides to establish connection between Datastage and Websphere MQ.
We will explain here step by step, how to configure Datastage and WebSphere MQ,
where in Datastage reads and writes message to/from a MQ queue. We use MQ
Connector stage (Datastage 8.5) for this purpose.

Prerequisites

We expect the following products installed and configured.

1. · IBM Websphere MQ V6 and above.
2. · Datastage server 8.5

If Websphere MQ and Datastage server are not installed in the same physical

machine, the server which has Datastage server must have MQ client installed.

Note: In the examples below, it assumed that the Websphere MQ and Datastage Server
are installed on different machines (nodes).

Content Task overview

•  Create MQ messages using IBM Websphere MQ
•  Websphere MQ connector Stage
•  Reading the MQ Message
•  Writing the MQ Messages

Create MQ messages

To create MQ messages, we require Websphere MQ explorer or similar tool.

After the Websphere MQ explorer got opened expand the queue manager’s folder

Provide the Queue manager name and click the next button till the below screen appears.

 
Give the “Listen on port number”. Give a new port number (should not be used by other
queues).

 
In the left pane of the Websphere MQ Explorer right click on folder
Queues -> New -> Local Queue

 
In the below screen give the Local Queue name and click Finish button.

A row entry with the Local queue name given is appeared in the right pane. Right click
on the row and click “Put Test Message…”

Give the data in the Message data and press “Put message” button for each entry.

To check the data in the Queue right click on the desired queue in the right pane and
press “Browse Message…”.

Websphere MQ connector Stage

The Websphere MQ connector Stage in Datastage read messages from and writes
messages to a Websphere MQ enterprise messaging system.
A queue manager manages one or more message queues. The MQ Connector stage
establishes a connection to a queue manager in order to read messages from or write
messages to a queue. To connect to the queue manager there are two types of mode.

Server mode: The queue manager resides on the same machine as the MQ Connector
stage.

Client mode: The queue manager resides on a remote machine Specify channel name,
transport type (TCP), and remote connection name or IP address.

Installation requirements must be met depending upon whether you use the Websphere
MQ Connector in server connection mode or in client connection mode.

This mode is determined by your selection in the Mode property.

Client connection mode

• The Websphere MQ client must be installed on the same node as the connector.
• The Websphere MQ server must be installed on a node in the same network in which the connector is installed. 
• The Websphere MQ server can be installed on the same node as the connector.
•  There must be a network connection between the client node and the server node.

Server connection mode

• The Websphere MQ server must be installed on the same node as the
• connector.

Reading the MQ Messages

Steps to read the data from the MQ to the sequential file in Datastage:
Create a new parallel job:
Draw the Websphere MQ Connector stage in the parallel job. This stage exists in the Real
Time section of the pallet box

Draw the output connector from the Websphere MQ Connector to a sequential file.

In the properties section of the Websphere MQ connector stage provide the following
details.

Mode: Client
Queue Manager: This name exists in the left pane of the Websphere MQ Explorer.
Username: Username for the server.
Password: Password for the server.
Channel name: Default Channel name
Transport Type: TCP
Connection name: Server IP (Listener port number given in the MQ explorer for the
Queue here 1416)
Queue Name: This name is found in the right pane of the MQ Explorer when the Queue
folder of the queue manager is clicked.

Click on the Columns Tab and enter the fields as below. Fields defined without specified
data elements are treated as payload fields.

If we want to display the MessageID as a column select WSMQ.MSGID from the Data
Element list.

After running the job, view the data of the sequential file. The test data given in the MQ

explorer appears. The fetched data is removed from the queue.

Writing the MQ Messages

Create new Local Queue (IBMTestQueue2) in the Queue manager (IBMTest)

In the Datastage parallel job, create an output link from a sequential file stage to a
Websphere MQ connector stage.

Provide some sample data in the sequential file.

In the properties table of the Websphere MQ connector (target) give the Queue manager
name, queue name and other information as below screenshot and press ok.

After running the job the data in the sequential file flows to MQ. To check the data in the
queue right click on the queue and press the “Browse message…” button.

Import WebSphere Modler Project to Process Server

Prerequisites

Following Software’s need to be installed in your local machine to run this application successfully.

1)     WBM 6.0.2.1

2)     WID 6.02 with Fix pack 5

3)     RSA 7.0

4)     WAS 6.1 Fix pack 5 in RSA 7.0

5)     Upgrade RSA 7.0 to RSA 7.0.0.3

Creating the Model in WBM

Creating a new Business Model project

To Create a Qual Modeling Project, Complete the Following Steps

         

1.     Open WebSphere Business Modeller with <<workspace>> in your local.

2.     Click on Cancel button on pop-up window when opening WebSphere Business Modeller.

3.     To create new project/model Open File > New Business Modeling Project

4.     In the Project Name Field, enter Qual.

5.     In the Default Process Catalog Name Field, enter QualProcessCatalog.

6.     Select the Check Box Field.

7.     In the Name Field, Enter QualProcess then click on Finish button.

Import All the Business Items (i.e. All Existing (.xsd) Files)

To Import the .xsd Files, Complete the Following Steps

To Import the .xsd Files, Complete the Following Steps

              

1.     Click “Apply 4-Pane Layout” icon in toolbar as shown below to view the “Project Tree” window.

2.     Right-click on Project (i.e. Qual) at “Project Tree” window.

3.     From the Pop-up menu click Import then select type as “Type Definition XML Schema (.xsd)” as shown below and click on next button.

4.     Accept the default as “Yes” in the pop-up window.

5.     Select “Source directory” which contains “.XSDs”  and select “Target project” as Qual from drop-down box in Source and destination tab click on “Finish” button (wait until all Business items generated properly)

6.     Accept “Ok” button for warnings as displayed below.

7.     Verify the Business Items in “Project Tree” window as like below.

Create a Business Process Model

1.     To create Business Process Model you need to change Modeling mode as “WebSphere Process Server” from Toolbar or Modeling option from menu bar as like below.

(OR)

2.     Delete the “Start Node” and “Stop Node” in “QualProces” window by right click on nodes as shown below and save using Ctrl+S.

                 

3.     Right click on Process window then select New -> Map and change the name as PrepareCP.

4.     Drag “task” icon from panel to process window as like below then change the name as “CallCP”.

5.     Right click on Process window then select New -> Map and change the name as PrepareOR.

6.     Drag “task” icon from panel to process window as like below then change the name as “CallOR”.

7.     Right click on Process window then select New -> Map and change the name as Prepare resp. Now your process window will display as like below.

8.     Generate the flow using “Create Connection” option from panel and link all the maps and tasks one by one as like below.

 

 

Assign Input/Output Of Each Components

               

                   Click each Input and Output for Component in Attributes –Input/Output -> -General tab.

1.     In Name Field is default it will be input or output. 

2.     In Associated Data Field, Click browse button and Select a Business Item. 

3.     All other Fields let default.  

  

4.     Select all the required Input and Output Business types. (Follow Step1, 2, 3).

5.     Save the Project.

Finally The Process will be like

Exporting From WBM

To export From WBM, Complete the following

                      

1.     Right-click on the Project and Click Export.

2.     Select Type as WebSphere Process Server from the pop-up window as shown below, click on next button.

3.     Enter the Target Directory name to export the Process into File System.

4.     Click on Finish button wait until export is done successful to display the pop-up window then click on OK button.

5.     Verify the Target Directory (TESTWBM1) as contains Process and Business Items as like below.



Importing into WID

Create Business Integration Module

1.     Open WID Workspace <<workspace>> and WID will open Business Integration Perspective.

2.    Right click on Business Integration window New -> Other -> Module.

3.    Enter Module Name as QS then click on Finish.

4.    Uncheck Build Automatically option form Project -> Build Automatically option.

5.    Enable Web Service developer capabilities from Window -> Preferences -> Workbench -> Capabilities -> select Web Service Developer check box then click on Ok.

6.    Go to Window > Preferences > Web Services > Code generation > IBM Webservice runtime) check the following options are selected, if not please select the check boxes and click on Apply and OK buttons.

Import WBM Process into WID.

1.     Right Click on Module QS -> Data Types select Import as shown below.

2.    Select “File System” in Import pop-up window then click on next button.

3.    Browse the location which contains “.XSDs” in From Directory  select all “.xsd” then click on Finish button wait until all Business Objects imported successfully.

4.     Right Click on Module QS select Import.

5.    Select “File System” in Import pop-up window then click on next button.

6.    Browse the location for “WBM Process File System location” in From Directory  select “qualprocess” folder then click on Finish button wait until process is imported into WID successfully.

7.     Now you can able to see Assembly Diagram as like below and save Assembly Diagram using Ctrl+S.

8.     Verify your problems console doesn’t contain any errors.

Modifications in WID

1.     Open QS -> Interfaces -> CallCP assign Input as “getCustomerProfileService” (Click on getCustomerProfileService -> Browse -> select “getCustomerProfileService” from Data Type Selection pop-up window -> ok).

2.     Assign Output as “getCustomerProfileServiceResponse”  (Click on getCustomerProfileServiceResponse -> Browse -> select “getCustomerProfileServiceResponse” from Data Type Selection pop-up window -> ok) and Save (Ctrl+S).

3.     Open QS -> Interfaces -> CallOR assign Input as “OrderRetrievalRequestType” (Click on OrderRetrievalRequestType -> Browse -> select “OrderRetrievalRequestType” from Data Type Selection pop-up window -> ok).

4.     Assign Output as “OrderRetrievalResponseType”  (Click on OrderRetrievalResponseType -> Browse -> select “OrderRetrievalResponseType” from Data Type Selection pop-up window -> ok) and Save (Ctrl+S).

5.     Open QS -> Interfaces -> QualProcess assign Input as “QualificationRequestType” (Click on QualificationRequestType -> Browse -> select “QualificationRequestType” from Data Type Selection pop-up window -> ok).

6.     Assign Output as “QualificationResponseType”  (Click on QualificationResponseType -> Browse -> select “QualificationResponseType” from Data Type Selection pop-up window -> ok) and Save (Ctrl+S).

7.     Open QS -> Business Logic -> Process -> QualProcess accept the pop-up window click on Ok button.

8.     Go to Properties -> Java Imports paste the below file contents.

9.     Configure the Variable data types, go to Variables -> QualificationRequestTypeVariable -> Properties -> Details -> Browse -> enter “QualificationRequestType” in Filter field select QualficationRequestType  from Matching Data types then click on Ok.

10.  Go to Variables -> getCustomerProfileServiceVariable -> Properties -> Details -> Browse -> enter “getCustomerProfileService” in Filter field select getCustomerProfileService from Matching Data types then click on Ok.

11.  Go to Variables -> getCustomerProfileServiceResponseVariable -> Properties -> Details -> Browse -> enter “getCustomerProfileServiceResponse” in Filter field select getCustomerProfileServiceResponse from Matching Data types then click on Ok.

12.  Go to Variables -> OrderRetrievalRequestTypeVariable -> Properties -> Details -> Browse -> enter “OrderRetrievalRequestType” in Filter field select OrderRetrievalRequestType from Matching Data types then click on Ok.

13.  Go to Variables -> OrderRetrievalResponseTypeVariable -> Properties -> Details -> Browse -> enter “OrderRetrievalResponseType” in Filter field select OrderRetrievalResponseType from Matching Data types then click on Ok.

14.  Go to Variables -> QualificationResponeTypeVariable -> Properties -> Details -> Browse -> enter “QualificationResponseType” in Filter field select QualificationResponseType from Matching Data types then click on Ok. Click on Save button.

15.  Then process will be displayed as like below.

16.  Change the “Prepare CP” Snippet as Assign using right click on “Prepare CP” select Change type -> Assign.

17.  “Prepare CP” activity changes as like below.

18.  Select “Prepare CP” goes to Properties -> Details, select Variable from “From” drop-down box, and select QualificationRequestTypeVariable -> requestHeader -> transactionIdentifier.

19.  Select Variable from “To” drop-down box, and select QualificationResponseTypeVaraible -> responseHeader -> transactionIdentifier then click on Save button. (Note when we are assigning From variable to To variable data type must match as same).

20.  Change the “Prepare OR” Snippet as Assign using right click on “Prepare OR” select Change type -> Assign.

21.  Select “Prepare OR” goes to Properties -> Details, select Variable from “From” drop-down box, and select QualificationRequestTypeVariable -> requestHeader -> transactionIdentifier..

22.  Select Variable from “To” drop-down box, and select QualificationResponseTypeVaraible -> responseHeader -> transactionIdentifier then click on Save button.

23.  Change the “Prepare resp” Snippet as Assign using right click on “Prepare resp” select Change type -> Assign.

24.  Add a new mapping relation using New button in Properties -> Detail window.

25.  Map the List of below Objects in Properties window of “Prepare resp” they are as like below:

a.     Customer Object : select From variable as getCustomerProfileServiceResponseVariable -> getCustomerProfileResponse -> customer, select To variable as QualificationResponseTypeVariable -> qualifyCustomerRespone -> customer.

b.    Add a new mapping relation using New button in Properties -> Detail window.

c.     ResponseHeader Object: select From variable as getCustomerProfileServiceResponseVariable -> getCustomerProfileResponse -> responseHeader, select To variable as QualificationResponseTypeVariable -> responseHeader.

d.    LineOfBusinessProfile Object: select From variable as getCustomerProfileServiceResponseVariable -> getCustomerProfileResponse -> lineOfBusinessProfile, select To variable as QualificationResponseTypeVariable -> qualifyCustomerRespone -> lineOfBusiness.

e.     CrossLineOfBusiness Object: select From variable as getCustomerProfileServiceResponseVariable -> getCustomerProfileResponse -> crossLineOfBusiness, select To variable as QualificationResponseTypeVariable -> qualifyCustomerRespone -> crossLineOfBusiness.

f.     ResponseMessageList Object: select From variable as getCustomerProfileServiceResponseVariable -> getCustomerProfileResponse -> responseMessageList, select To variable as QualificationResponseTypeVariable -> responseMessageList.

g.    OrderDetail Object: select From variable as OrderRetrievalResponseTypeVariable -> orderDetailList -> orderDetails, select To variable as QualificationResponseTypeVariable -> qualifyOrderResponse -> orderDetailList -> orderDetail and click on Save button.

26.  Add a Snippet between CallOR and “Prepare resp” by select snippet activity.

27.  Change the link from CallOR to Snippet, and Snippet to “Prepare resp”.

Before      After

28.  To arrange the contents properly right click on QualProcess -> select “Align Parallel Activities Contents Automatically”.

29.  Select Snippet -> Properties -> Details -> Select radio button as Java then accepts “Yes” button paste the below file contents in TextArea then click on Save button.

30.  Start WebSphere Process Server 6.0 in Servers tab, go to Servers tab select WebSphere Process Server6.0 -> Right Click and select Start (Wait until Server started successfully)