Exposing IBM DataStage Jobs as WebService

1. Introduction

Business analysts and subject matter experts can use IBM^® InfoSphere™ Information Analyzer to manage the Information Services in their IBM Information Server (IIS).
IBM® InfoSphere™ Information Analyzer evaluates the content and structure of your data for consistency and quality. InfoSphere Information Analyzer also helps you to improve the accuracy of your data by making inferences and identifying anomalies.
InfoSphere Information Analyzer helps you to assess the quality of your data by identifying inconsistencies, redundancies, and anomalies in your data at the column, table, and cross-table level. InfoSphere Information Analyzer also makes inferences about the best choices regarding data structure. Inferences help you to learn more about the optimal structure of your data and what you can do to improve the quality of your data.

1.1 Information Analyzer Feature overview

• Import metadata from various data environments
• Configure system analysis options
• Create virtual columns from physical columns
• Analyze column data classification
• Analyze column data properties
• Analyze column data completeness and validity
• Analyze column data formats
• Analyze data value commonality across columns
• Analyze table primary keys
• Analyze duplicate primary key values
• Analyze table foreign keys
• Analyze referential integrity
• Create analytical notes to supplement system results
• Capture enterprise data management (EDM) data to supplement system results
• Produce system reports to display system results

This article focuses on setting up the IIS environment to generate WebService reference for the DataStage Jobs. By the end of this article you will see exposed WebServices for the DataStage ETL Jobs and the method of calling these exposed WebServices from Standalone java application.

            The objective of this article is to explain step by step the process of exposing the ETL Jobs as WebService Interface. So it’s not explaining the process of installation/configuration of the product stack or the process of developing the ETL Jobs.

2. Solution Scenario

            The sample application is based around customer profitability and churn management for Telecom industry. We have existing ETL jobs.

So we have following products already installed and configured.

• DataStage Server v8.1 (https://w3-103.ibm.com/software/xl/download/ticket.do, Part number: C1W3PML).
• Fixpack for DataStage Server v8.1

(http://iidemo1.svl.ibm.com/files/IF131646.zip ).

• VC++ Express edition

(http://www.microsoft.com/express/download/#webInstall).

• Platform SDK for windows (http://www.microsoft.com/downloads/details.aspx?FamilyId=A55B6B43-E24F-4EA3-A93E-40C0EC4F68E5&displaylang=en).

3. Configuration

Open the Information Analyzer from Start -> All Programs -> IBM Information Server -> IBM Information Server Console (or) Console for IBM Information Server icon at Desktop with provided valid User credentials (Suite Administrator, ex: isadmin/inf0server).

3.1 Create Information Service Connection

            Information Service Connection is used for Information Service Application to list down all ETL Jobs, User can select Jobs to create operations at Information Service Application.

1. Open the Information Service Connection at Home -> Configuration -> Information Service Connections.
2. Click “New” under Tasks, enter mandatory fields (Connection Name, Information Provider Type) as like below
3. On selection of Information Provider Type as “DataStage and QualityStage” from combo box, it will ask to fill the Agent Host, Username, and Password details.  Click on Test when the details filled.

                              

4. Click on “Save, Enable and Close” button at the bottom of the page.
5. On successful creation of Information Service connection User can see entry of Information Server Connection as like below.

4. Information Services

            Information Analyzer is client server application, User can create two kinds of projects they are Information Services and Information Analyzer. 

Information Service project is used to create the services/operations for the InfoSphere entities (Jobs, Database Queries etc).

Information Analyzer project is used to analyze the data in many ways example like Column Analysis, Cross Domain Analysis, Primary Key Analysis and etc.

Create an Information Service project to develop the Web services to access the DataStage Jobs from any language (ex: Java, .Net, etc.).

1. Click “New Project” button, enter the Project name as “CISProject”, select type as “Information Services”, click Ok button.

2. Go to Develop -> click on Information Services Application

3. Create a new Application using the “New” link under Tasks.
4. Enter the name, description at Overview tab.
5. Click “New” button at bottom of the “Select a View” pane at left side of the page then click on “New Service” button.

                    

              

6. In New Service window, user has to provide “Service Name” as “CISService” and description.

              

7. Click on “New Operation” button at bottom and enter the Operation details.

                      

8. Click Select button of “Select an information provider” at Information Provider tab of Operation.
9. Select Type as “DataStage and QualityStage” -> display the Information Service Connection at “Select a DataStage Job” section, double click on Connection -> double click on Project -> double click on Jobs folder -> select your Job -> Click on Ok button.

        

10. Repeat the steps 7 to 9 to add new operations for All Jobs.
11. Go to Bindings section -> select “SOAP Over HTTP” at Attach Bindings section.
12. Click on save application and click on Close application.
13. Click on Deploy from Tasks at right side of the screen of the CIS Application.
14. Wait until application deployed successfully, user can see the Activity Status window at bottom, once displayed as “Completed” means service is deployed successfully.
15. Open “Deployed Information Services Applications” under Operate.  Click on CISApplication -> click on CISService -> click “View Service in Catalog” button.

                            

16. Service details will display in default browser, go to Bindings section -> Extract SOAP Over HTTP Section -> Click on “Open WSDL Document” -> Save Document with .wsdl format.

4.1 Calling Information Services as Ant Task

1. Create a Java Project using your favorite IDE to create a WebService Client application for the saved .wsdl file
2. Copy the .wsdl file into Java Project; right click on Wsdl file -> Web Services -> Generate Client.

        

3. Select all default options to generate the WebService client.
4. User can see the generated port, service information as like below.

5. Open the wsdl file in text editor; right click on wsdl file -> Open with -> Text Editor.
6. Locate the tag <wsdlsoap:address> in the file, copy the location of URL to test in browser.

7. As active service will display the message as like below.

8.      Write a sample code to test the Information Service from Standalone Java application. (code is attached below)

                       

9. Run CISService application as Java Application with argument as “AllocationType”.

10. User can see the DataStage Jobs running status at “Websphere DataStage and QualityStage Director” as like below.

After successful running status changed from Running to Finished

11. User can see the exceptions, warnings, and information for the Jobs at Director by right click on the Job -> click on “View Log”.

                 

12. Log description will displays as Log view in Director, to see the detailed information for the log double click on the log entry.

13. Export Project as .jar file used as library API for Ant Tasks.
14. Write Ant Task for the Jobs to execute using build.xml, include the below code.

15. Execute ETL Jobs using ant task at Command prompt using the ant command as “ant –f build.xml runETLJobs”