HP NonStop Data Transformation Engine Intelligent Business Integration Reference Guide Abstract This document details the components of the HP NonStop™ Data Transformation Engine (NonStop DTE) and Mercator product architecture. Product Version NonStop Data Transformation Engine 6.7.
Document History Part Number Product Version Published 528258-001 NonStop Data Transformation Engine 6.7.1 June 2004 528258-002 NonStop Data Transformation Engine 6.7.
Contents About This Document Related References........................................................................................... 5 Chapter 1 - Introduction Mercator Technology......................................................................................... 7 Design Environment – Design Studio....................................................................... 8 Type Designer ............................................................................................ 9 Map Designer ....
Contents Chapter 7 - Securing HTTP Communications What is SSL?................................................................................................... 64 Encryption ............................................................................................... 64 Digital Signatures and Certificates .................................................................. 65 Implementing an SSL Environment ....................................................................... 65 Event Agent............
About This Document This document details the components of Mercator’s product architecture. This document also details Mercator’s transformation, connectivity, flexibility, and scalability capabilities, demonstrating how Mercator products can be the robust backbone of your critical business operations. Related References The Mercator Online Library contains information about all products. Refer to the following references for more information about topics frequently discussed in this document.
About This Document Related References Related Reference Description Map Designer Reference Guide The Map Designer user interface and instruction on specifying mapping rules, configuring map sources and targets, defining map-level settings, and executing a compiled map. Mercator Integration for Java Reference Guide How to use the components included with the Mercator Integration for Java product, which allow for server-independent map execution.
Chapter 1 - Introduction Mercator Technology Mercator products combine leading-edge technology for both external (B2B) and internal (A2A) application integration, powerful graphical technology for integrating application content, and management facilities that provide run-time control of both real-time and event-driven integration activities. They support the entire integration solution life cycle, beginning with graphical definition of integration workflows.
Chapter 1 - Introduction Design Environment – Design Studio Design Environment – Design Studio The Mercator Design Studio is the client interface into an extensive array of functionality that is available to create application integration solutions. Application integration solutions created with the Design Studio are executed on transformation servers specific to the Mercator product being used.
Chapter 1 - Introduction Design Environment – Design Studio Type Designer The Type Designer is a facility for creating, maintaining, and viewing data object definitions in a graphical form called a type tree. The type tree provides visual cues for understanding the structure of an object and supports point-and-click techniques enabling you to “drill down” to uncover an object’s complete definition.
Chapter 1 - Introduction Design Environment – Design Studio Type trees can be created in a number of ways: ♦ The Type Designer includes an importer tool for automatically generating type trees for data described in XML, COBOL, SAP R/3, Tuxedo, and PeopleSoft formats or application data structures. ♦ The Type Designer enables creation of type trees “from scratch” as well as modification and customization of existing type trees; including those generated by importers, DTD, or provided by Mercator.
Chapter 1 - Introduction Design Environment – Design Studio The type definitions in a type tree can be exported to a flat file in XML format for manipulation by XML and text-based tools. The modified trees can then be imported back into the Type Designer. Mercator provides the DTDs for the importing and exporting of type trees. Below is a portion of a document file generated by exporting a type tree: Note Refer to the Type Designer Reference Guide for more information.
Chapter 1 - Introduction Design Environment – Design Studio Type Tree Differences The type tree differences feature enables the user to compare two type tree files. When the analysis is complete, both trees display. Source compare differences will appear with either blue or red text. When a type is present in one tree but not present in the other tree, the text will be blue. When a type is present in both trees but have different properties, components, or restrictions, the text will be red.
Chapter 1 - Introduction Design Environment – Design Studio Map Designer The Map Designer is used to create maps, which consist of rules for data content, transformation, and intelligent routing. This process is facilitated by convenient “from” and “to” windows, drag-and-drop techniques, and spreadsheet-like rules. Mapping rules are added using logical statements.
Chapter 1 - Introduction Design Environment – Design Studio Input objects are not consumed when used in a transformation rule. This allows for a one-to-many relationship between an input and its outputs. An output can be constructed from any number of zero or more inputs. This creates a one-to-many relationship between an output and its inputs. Taken together, Mercator products provide many-to-many transformation support between inputs and outputs.
Chapter 1 - Introduction Design Environment – Design Studio As with the Type Designer, the Map Designer is able to export its rules and execution settings to a flat file in XML format. Note Refer to the Map Designer Reference Guide for more information.
Chapter 1 - Introduction Design Environment – Design Studio Map Source File Differences As with the Type Designer, the Map Designer also has a graphical source compare feature that enables you to compare two map source files.
Chapter 1 - Introduction Design Environment – Design Studio Integration Flow Designer The Integration Flow Designer (IFD) is a graphical manager for data flows. It is used during development to define a set of logically related integration solutions called a system. The IFD defines interactions among maps and systems of maps, where maps are objects that serve one or more purposes such as routing or transformation.
Chapter 1 - Introduction Design Environment – Design Studio ♦ Build and port entire systems with simple mouse-clicks Note Refer to the Integration Flow Designer Reference Guide for more information. Graphical Design of Data Flow Visual data flows are created to aid in the design of integration solutions. A graphical palette enables you to produce system definition diagrams and navigate easily among the Mercator Design Studio tools.
Chapter 1 - Introduction Design Environment – Design Studio Parameters for different servers representing different operating environments can be defined within the IFD. A server can then be assigned to a particular system of maps, therefore becoming the “active” server for this system.
Chapter 1 - Introduction Design Environment – Design Studio After the appropriate server definitions have been created, it is then possible to produce, using the wizard, automated deployment definitions that are specific to each server. It is possible to specify within a script, that all maps are built and transferred, or only a specific subset of the maps in a system. Additional files required to support the system’s operation can also be included in the transfer.
Chapter 1 - Introduction Design Environment – Design Studio The Database Interface Designer makes it possible to: ♦ Specify databases, user IDs, passwords to connect to as sources and destinations ♦ Define query statements ♦ Generate type trees automatically for queries, stored procedure input and output parameters or tables ♦ Define specific columns to update for update transactions ♦ Define tables for which inserts, updates, and deletions will trigger maps via the Mercator Event Server Note Refer to th
Chapter 1 - Introduction Design Environment – Design Studio Source Control All of the Mercator Design Studio components provide a direct interface to any source control package that supports the Microsoft Source Code Control Interface API. This means that source components in all of the designers can be directly managed within a source control package (such as Merant PVCS, Rational ClearCase, and Microsoft Visual SourceSafe).
Chapter 2 - Integration Maps The core components for any Mercator product-based application integration solution are the maps.
Chapter 2 - Integration Maps Anatomy of a Mercator Map Anatomy of a Mercator Map A map is comprised of the following components: ♦ Input and output map cards that represent the sources and targets. ♦ Map rules that define how the output data is built based on business rules. ♦ Specification of resource adapter used to retrieve and send data.
Chapter 2 - Integration Maps Anatomy of a Mercator Map Input and Output Cards Each input and output is represented in a separate map card. The data structure for each of the inputs and outputs is represented in the form of a type tree created using the Type Designer.
Chapter 2 - Integration Maps Anatomy of a Mercator Map Adapter Configuration for Input and Output Mercator resource adapters provide the connectivity needed to GET data from sources and PUT data to targets. However, adapters do not provide transformation; that is the purpose of map rules. Map rules describe the nature of the transformation while the transformation server performs the actual transformation of the data. Mercator provides adapters for common enterprise-level resources.
Chapter 2 - Integration Maps Anatomy of a Mercator Map Note Adapter settings defined in the map can be overridden with different adapter settings at map runtime. The resource adapters are loosely coupled with the transformation process. An adapter retrieves data to be processed by the map as a byte stream. This means that to the extent that the type definitions and byte streams are compatible, type trees and adapters can be used interchangeably.
Chapter 2 - Integration Maps Map Behavior During Execution Map Behavior During Execution The execution characteristics of a map are determined by the execution parameters that are configured at design time. However, these map execution characteristics or map settings can also be overridden at run time. The behavior of a single map invocation is as follows: 1 The adapters configured for their respective input cards retrieve some of the input data.
Chapter 2 - Integration Maps Map Behavior During Execution Input and Output Card Settings Card settings define the data object the card represents, and how the data is retrieved or routed, and compiled into the map. The behavior of input and output cards configured in the compiled map can be overridden. Schema Setting This setting defines which type tree definition is to be used to represent the data source for an input card and the output data structure for an output card.
Chapter 2 - Integration Maps Map Behavior During Execution SourceRule Settings SourceRule settings in the input card specify how and what data to retrieve, where to get it, how much to get, and what to do when an error occurs. The Source card setting specifies the adapter source of the input data.
Chapter 2 - Integration Maps Map Behavior During Execution FetchAs Setting This setting applies to input cards only. It dictates how the adapter retrieves the input data. By default, this setting is Integral. This means that all of the input data is retrieved once from the source. Setting this parameter to Burst means that the adapter will retrieve the input data in increments specified in the FetchUnit parameter. The units of data correspond to the highest-level repeating object in the type tree.
Chapter 2 - Integration Maps Map Behavior During Execution There are a number of potential benefits in using Burst mode: ♦ A single map consumes a large quantity of input data, but only a small portion of that data is needed to build an individual output. Burst mode can be used to minimize the size of the workspace (an internal index of types present in the data stream). In some cases, a smaller workspace can increase map performance. ♦ Coordinating multiple inputs together.
Chapter 2 - Integration Maps Map Behavior During Execution Map Settings The previous section outlined the settings that apply to individual input and output cards. However, there are also map settings that dictate the behavior of the entire map, across all input and output cards.
Chapter 2 - Integration Maps Map Behavior During Execution The MapAudit setting used in combination with the WorkSpace setting determines whether different threads of the same map component will run concurrently (multi-threaded) or sequentially (single-threaded). It is suggested to create a trace only during development as performance can be adversely affected. The WorkSpace settings specify memory page settings enabling you to optimize the speed of map processing.
Chapter 2 - Integration Maps Map Behavior During Execution Note As mentioned in the Adapter Configuration for Input and Output section, virtually all of the source, target, and map setting parameters can be overridden with different values at runtime. This capability is available for all modes of map execution.
Chapter 2 - Integration Maps Map Behavior During Execution The Resource Registry provides a Java-based graphical user interface (GUI) for creating and managing these resource definitions. Step 1: A resource name is created. Step 2: A resource name is used to identify the deployment environment’s resources. The different values of the resource name are resolved at run-time by the map server. Step 3: The resource’s parameters are defined for multiple deployment environments.
Chapter 3 - Methods of Execution The core execution engine in Mercator products is developed in ANSI C for platform portability and high performance. However, there are a variety of mechanisms that can be used to invoke integration components, including a Java API that enables Java-based interfaces to maps. Other Java-based mechanisms for map execution are servlets and Enterprise JavaBeans (EJB).
Chapter 3 - Methods of Execution Servlet Execution Model This approach has been successfully employed in a number of implementations, in particular, to provide the capability to Web-enable legacy applications. The powerful transformation and connectivity capabilities of Mercator products provide the application connectivity. Utilization of the automatically generated servlet exposes the power of this application integration capability to the Web.
Chapter 3 - Methods of Execution Enterprise JavaBean Execution Model ♦ Overriding input and output map cards and sending and returning data using HTTP ♦ Enabling and overriding map runtime settings ♦ Generating and compiling servlets The Servlet Integrator wizard provides you with step-by-step instructions on how to create, set up, and execute a Java servlet.
Chapter 3 - Methods of Execution Command-Driven Execution Model Command-Driven Execution Model The Mercator Command Server provides a command-driven capability for the execution of integration maps. This model is normally associated with batchdriven, one-off integration processes.
Chapter 3 - Methods of Execution Command-Driven Execution Model Execution Command Overriding a Database Target Example Suppose you want to run the map multitran.hp, overriding some of the settings for the fourth output card, which is a database target. Use the following command: /mercator/bin/mercator /Mercator/maps/multitran.hp –G –OD4R5:3B ‘–TRACE ERROR’ Option Description multitran.hp Specifies the name of the map to be executed.
Chapter 3 - Methods of Execution Event-Driven Execution Model Event-Driven Execution Model The event-driven execution model relies upon the Mercator Event Server to coordinate the triggering of map threads based upon either input event or time event triggers. Event Server Architecture The Event Server architecture is an event-driven software model executing maps. Systems contain maps supporting a system data flow that is specified by using the Integration Flow Designer.
Chapter 3 - Methods of Execution Event-Driven Execution Model A watch may have multiple map instances executing simultaneously. The Event Server uses its resource manager to insure that map inputs and outputs remain in a consistent state.
Chapter 3 - Methods of Execution Event-Driven Execution Model Event Server System Example The following is an example of a system defined in the Integration Flow Designer and executable by the Event Server: Intelligent Business Integration Reference Guide 44
Chapter 3 - Methods of Execution Event-Driven Execution Model The data flow for this system is as follows: 1 The VBInput source to the VBtoSQL map component is a file. The arrival or update of this file is the trigger for the automatic invocation of an instance of the VBToSQL map component by the Event Server. This trigger, represented ), displays on the VBtoSQL map component icon by the glasses symbol ( and on the VBInput card.
Chapter 3 - Methods of Execution Event-Driven Execution Model Audit and Transaction Logging Example The totally flexible architecture of Mercator products allows the design and development of sophisticated transaction and audit logging using standard functionality. This is supported by the complete range of resource adapters and the many tools available in the complete line of Mercator products.
Chapter 3 - Methods of Execution Event-Driven Execution Model ♦ The output from ErrorLog is delivered to a file. Each time a map fails, an entry is appended to this file. An alternative to this file-based method is to use the Mercator SNMP Agent to issue traps, which trigger alerts in SNMP-enabled network management software (such as HP OpenView, Tivoli NetView, and BMC Patrol). ♦ Every single map thread execution is logged to an execution history database through the ExecutionHistoryDB output.
Chapter 3 - Methods of Execution Event-Driven Execution Model Because an application or Web server is not required, complete A2A, B2B, and B2C capabilities are provided without the need to resort to programming. The Event Agent supports HTTP/1.1, and with the installation of the Security Option, supports the following protocols: ♦ Secure Sockets Layer 2.0 (SSLv2) ♦ Secure Sockets Layer 3.0 (SSLv3) ♦ Transport Layer Security 1.
Chapter 3 - Methods of Execution Event-Driven Execution Model Event Agent Operation The following is an example of how the Event Agent, HTTP adapter, and the Event Server work together to process HTTP requests: The processing sequence in this example is as follows: 1 When the Event Server is started, Map A, which is expecting an HTTP request as input, sends an HTTP request to the Event Agent via the HTTP adapter.
Chapter 3 - Methods of Execution Event-Driven Execution Model 4 The map triggered by the HTTP request generates its output, which is passed onto another map whose output may be passed onto another map, and so forth. In all cases, the session ID assigned by the Event Agent is passed along to each subsequent map in the form of a wildcard parameter.
Chapter 3 - Methods of Execution Application-Embedded Execution Model Application-Embedded Execution Model The Mercator API provides the ability to execute maps programmatically as “embedded” processes and is included in the Mercator Software Development Kit (SDK). The API, available for both the C and Java languages, interfaces directly with the core execution engine in Mercator products running on the appropriate target platform.
Chapter 4 - Developing Custom Adapters The interface for developing custom adapters makes it possible to develop adapters with as much functionality as those provided with the product.
Chapter 4 - Developing Custom Adapters XML Descriptor File XML Descriptor File The XML descriptor file adapters.xml, which is located in the Mercator installation directory, contains a single tag per adapter, including a number of attributes to provide information about the adapter. The following is an excerpt from the file: PAGE 54Chapter 4 - Developing Custom Adapters XML Descriptor File Attribute Description library Name of the DLL or shared library. The DLL, SO, or SL file extension should not be provided. In addition, for UNIX adapters, the name should not be prefixed with lib. This will be added when accessing the adapter. vendor Name of the adapter manufacturer. The vendor name is displayed along with the adapter name in the Map Designer and Integration Flow Designer.
Chapter 4 - Developing Custom Adapters 3 XML Descriptor File The adapter is now a selection in the adapter list in the Map Designer and Integration Flow Designer.
Chapter 5 - Operation Management & Monitoring Capabilities Mercator products provide a number of capabilities and components for managing and monitoring the integration execution environment such as: ♦ A Management Console that provides detailed monitoring information as well as dynamic configuration features for systems under control of the Event Server. ♦ Ability through SNMP to provide the same detailed monitoring information to third-party management software.
Chapter 5 - Operation Management & Monitoring Capabilities Management Console It also permits centralized, remote management of all Event Server installations. Separate connections are defined for each Event Server installation.
Chapter 5 - Operation Management & Monitoring Capabilities Management Console From the Management Console, it is possible to remotely stop, start, and pause systems. Event Server debugging can also be enabled to assist in operational troubleshooting of Event Server operations. The Management Console provides a variety of statistics about the maps being run by the Event Server, such as how long the Event Server has been running, how many maps have been processed, and so on.
Chapter 5 - Operation Management & Monitoring Capabilities Management Console Event Server Administration An Event Server Administration service is available to configure the systems that can be controlled by the Management Console. The Event Server Administration enables the configuration of secure access rights to monitor and control individual systems running under one or more Event Servers.
Chapter 5 - Operation Management & Monitoring Capabilities SNMP Support SNMP Support With the Mercator SNMP Agent, you can monitor and manage the status of the Event Server and associated map events.
Chapter 6 - High-Availability and Load-Balancing Architecture The following are the high-availability and load-balancing options that Mercator products currently offer: ♦ Server clustering ♦ Enterprise JavaBeans Server Clustering A server clustering approach caters for high-availability and failover but does not address the load-balancing requirement. The Mercator products support failover in the Microsoft Cluster Service and Sun Cluster 3 server environments.
Chapter 6 - High-Availability and Load-Balancing Architecture Server Clustering The following diagram illustrates a Windows cluster environment for the Event Server. Clustering Support for the Event Server " " The Integration Broker is installed on each node that is part of the cluster network. Each computer must have MSCS running on Windows 2000 Advanced Server. MSCS Node MSCS Node Computer 1 Computer 2 Becomes active upon failure of Node 1. No " Only one node is active at a time.
Chapter 6 - High-Availability and Load-Balancing Architecture Enterprise JavaBeans Enterprise JavaBeans By utilizing an EJB integration approach, maps can be wrapped inside both stateful and stateless Session beans, and deployed to EJB 1.0, EJB 1.1, and J2EEcompliant application servers. The application server environments themselves provide extensive load-balancing and failover between redundant instances of EJBs running on different physical application servers.
Chapter 7 - Securing HTTP Communications The two main requirements of a secure environment are the Secure Socket Layer (SSL) protocol and digital certificates. To activate the SSL capabilities within your Web server, all you need to do is install a digital certificate on your server. An SSL server secures an HTTP session by encrypting the information exchanged between a client and the server.
Chapter 7 - Securing HTTP Communications Implementing an SSL Environment Digital Signatures and Certificates A digital signature is used to sign or authenticate a message. A digital signature uses a private key to encrypt a portion of the message. The corresponding public key, which is given to everyone, is used to decrypt the message.
Chapter 7 - Securing HTTP Communications Implementing an SSL Environment The utility RSA BSAFE SSL-C from RSA Security Inc. and included in the Security Option install, enables you to generate a self-signed digital certificate installable on the Event Agent. A self-signed certificate is a digital certificate that is not issued and signed by a CA. This type of certificate can be used when testing the SSL capabilities of the Event Agent.
Chapter 8 - Web Services Mercator software provides integration solutions that accelerate the adoption of web services, eliminating the expensive, time-consuming, and risky overhaul of critical business systems. By combining the powerful integration capabilities of the Mercator Integration Broker with the emerging web services standards, Mercator products make it possible for organizations to provide and employ web services at both the business process and data process levels.
Chapter 8 - Web Services Technologies Used in Web Services If the answer to any of the above questions is yes, then using Mercator Integration Broker products such as our SOAP and HTTP adapters, the Event Server, and the Event Agent, which acts as a listener for web service requests, is the way to go.
Chapter 8 - Web Services Technologies Used in Web Services A WSDL document is comprised of a number of elements: ♦ Type definitions for data elements (normally using XML Schema) ♦ Message definitions that comprise one or more typed data elements ♦ Operation definitions that are abstract descriptions of actions supported by the service, and which define what are the input and output messages ♦ Port Type definition lists the set of operations supported by the Web Service ♦ Binding definition describes the b
Chapter 8 - Web Services Web Services Architecture The UDDI invocation model is as follows: 1 You use the UDDI business registry (either via a Web interface or using the inquiry API) to locate the business entity information for the business partner advertising the web service. 2 You drill down to find the advertised web services and obtain the binding templates (WSDL documents). 3 You prepare the program based on the WSDL specifications.
Chapter 8 - Web Services Web Services Architecture Consumer Architecture For design time purposes, the WSDL Importer generates type trees from WSDL files. WSDL Importer WSDL file Type tree Type Designer For run time purposes, the SOAP adapter is used to invoke the web service by sending a request message and processing the response. Note that two adapters are used in communicating between the map and the web service: the SOAP adapter and the HTTP adapter.
Chapter 8 - Web Services Web Services Architecture Provider Architecture Mercator Integration Broker, acting as a provider of web services, receives SOAP requests over HTTP, executes a map system, and then sends a SOAP response. SOAP Request Consumer SOAP Response Event Agent HTTP Adapter SOAP Adapter Map SOAP Adapter HTTP Adapter Event Server Because the map or system performs a process over the Web, it is considered a provider of a Web service.
Chapter 8 - Web Services Web Services Example Web Services Example The install_dir\integration_package\WebServices_v1\examples directory contains sample files to be used with Mercator Web Services components, including the WSDL Importer, SOAP and HTTP adapters, Event Agent, and the Event Server, to implement web services strategies. There are three examples: 1 Provider - This example demonstrates how to provide a web service using an existing map and exposing it as a web service through a wrapper map.
Index A E adapters customizing, 52 FTP, 65 HTTP, 47 overview, 26 SOAP, 68 adapters.
Index L building and deploying systems, 18 creating a data flow, 18 logic analysis, 18 overview, 17 O output maps, 25 settings, 32 L R load-balancing EJBs, 63 server clustering, 61 logic analyzing in the Integration Flow Designer, 18 related references, 5 resource aliases, 35 Resource Registry, 35 S security for HTTP messages, 64 server clustering, 61 Servlet Integrator overview, 37 SNMP Agent, 60 SOAP, 68 adapter, 68 source control, 22 SSL, 64 implementing with Event Agent, 65 implementing with Mer
Index W UDDI, 69 using maps to invoke, 70 WSDL, 68 Web-based execution Enterprise JavaBean, 39 HTTP processing requests from, 47 Servlet Integrator, 37 WSDL, 68 WSDL importer, 69 W web services, 67 consuming, 71 definition, 67 examples, 73 providing, 72 SOAP, 68 technologies used in, 68 Intelligent Business Integration Reference Guide 76
