Ibm Infosphere Information Server

Published Date: 02 Nov 2017

Information has been a learning source. But it can be more an encumbrance and not a profit, if it is not organized properly, or processed, and made usable to the concerned user in a form for decision making,

Over the years, IT industries have arisen into composite systems and this disunited surroundings has resulted into major problems and in turn are breaching business processes, such as hindering Customer Relationship Management (CRM), Enterprise Resource Planning (ERP), and Supply Chain Management (SCM), perverting analytics, and charging great deal of billions a year.

Master Data is the critical business information corroborating the transactional and analytical functioning of the enterprise. Master Data Management (MDM) is a collection of applications and technologies that strengthens the incorporated master data, and contemporizes it with analytical instruments. It makes significant benefits in decision-making.

MDM approaches the enterprise data quality job at its beginning point on the operational slope of the business. Hence data warehousing / analytical side of the business are aligned together and is very productive in leading enterprise across the globe.

Master Data Management (MDM) is the comprehensive stage for organizations to increase time to value, improve information quality and decrease integration revenue to the company. Information management also stresses on planning and implementing solutions throughout the enterprise, and assists to gain the in process governance. It cedes most trusted information throughout the entire information supply chain which boosts the business process by gaining organizations perceptivities.

1.1 OVERVIEW

IBM infosphere is a platform for trusted information, data integration, data ware housing, master data management, big data and information governance. Its just like a foundation for enterprise having large no of projects which provides them various facilities like performance scalability, reliability and speeds up the proceess of delivering quality data and face the challenges.The products used are: IBM Information Server and Master Data Management

1.11 IBM Infosphere Information Server:

IBM InfoSphere Information Server is the platform which provides context rich and trust worthy information by understanding, extracting, cleansing, and performing various transformation on the input data and can scale to large extent and meets any kind of volume recquiremnt. Information server provides faster result with high quality of data

The Functions of Information Server:

Understand The Data :

Infosphere Server provides facilities to import data from various sources and helps in analyses of the information by data profiling, and anlysis of relationships, hence it decreses the hazards, Information server hence provides increased productivity by data quality auditing

Cleansing The Information

InfoSphere Information Server cleanses the data by removing unwanted information, redundant information, applying some standardizing, to produce data with consistency and high quality. Information server also helps in creating single and error free information across the organisation.

Transform The Data Into Information

InfoSphere Information Server transforms the fetched data into a form useful for the next process. There are different types of existing transformation functions are present which provides processed data

Deliver The Information:

Information Server has ability to provide information in different forms, in different volumes, depending upon time, event, from one place to other, or at one place.

It also synchronizes information within itself.

IBM Infosphere Master Data Management

Master data is the most critical business data regarding the people, organisation, household so on, that is stored and communicated across the enterprise. It has very high value and plays a major role in all the business deals, and important decision.

MDM is a platform and a paradigm that provides the enterprise with trusted information and a 360 degree view of information from the data input of various sources which can be either a database, a flat file, for the matter of fact even unstructured data. It has set of process of operations on the information and disciplines that results into consistent data and relationship between the entities. MDM helps the organisation to govern the flow of bussiness information in very accurate manner. MDM server centralises the process and enables to acquire data with the help of various bussiness services built, that represents entire MDM platform.

The objective of the project is to develop an MDM Input/Output stage so that MDM can be seamlessly integrated into the IBM InfoSphere Information Server ETL job flow. This integration is achieved by automating couple of steps performed by the MDM. These steps need to be done at Infosphere Information server and it should convert the data into the form as required by the MDM automatically and pass that data to MDM for processing. In this way connectivity is achieved between Infosphere Information Server and Infosphere MDM.

1.2 Literature survey

Literature survey is mainly carried out in order to analyze the background of the

current project which helps to find out flaws in the existing system & guides on which unsolved problems can work out. So, the following topics not only illustrate the background of the project but also uncover the problems and flaws which motivated to propose solutions and work on this project. The purpose of this study is to provide background information on the issues to be considered in this thesis and to emphasize the relevance of the present study.

In order to ascertain data quality the master data architecture controls divided access, reverberation, and data flow which is one among the part of MDM processing. The research in the field of knowledge about mdm architecture is not that comprehensive. Earlier contributions in this field can be found in data mining and data warehousing, in which they considered mdm architecture as major thing in standardization and exact reporting dimensions.

In the mid1990s researchers discussed intensively the concept of "Planning the Strategic Data" which result into a data architecture. Albeit not using the term "master data" Planning the Strategic Data refers to information that is used on a company-wide level. More recent studies addressing the topic of the master data architecture have been a case study on multidimensional databases and an investigation of the product information supply chain in the consumer goods industry. Furthermore, Allemang made the distinction between linked data enterprise architecture and master data architecture without specifying the latter any further.

The most comprehensive scientific contribution to the topic has been made by Otto and Schmidt (2010). They refer to master data architectures as information architectures the scope of which is restricted to master data. These authors use the notion of "infor- mation architecture" not only to include shared data sources and data flows between data bases into the concept, but also to emphasize the demand for "authoritative" data standards and definitions. In their under- standing, the master data architecture comprises both a conceptual master data model and the application architecture for master data. The application architecture for master data can be further divided into source and target systems of master data on the one hand and data flows on the other hand. Moreover, the authors identify a set of design decisions related to the master data architecture.

Many contributions can be found in the research community addressing the general data architecture. There are a number of architecture models —the Zachman Frame- work, The Open Group Architecture Framework (TOGAF) , Enterprise Architecture Planning (EAP) , and the Enterprise Architecture Cube, just to name a few—consider the data architecture to be part of the overall enterprise architecture. Otto and Schmidt (2010), however, show that these frameworks by definition are not useful in providing both the conceptual breadth and depth required to investigate, analyze and design a master data architecture in detail. Master data architecture is a topic of discussion also in the practitioners’ community.

A prominent aspect in this regard is the question for the right architectural style. Analyst company Gartner, for example, distinguishes between three styles . Design/construct MDM is needed if a new business is created, operational MDM supports regular operations of existing businesses, and analytical MDM is mainly used for reporting purposes. Oberhofer and Dreibelbis (2008) propose a similar categorization, describing also operational and analytical approaches, but identifying also other type of collaborative method. In these different styles of architecture it relates to the flow of data between a golden record of organisation and data source systems. An analytical architecture, for example, uses a unidirectional flow of data from the source systems to the "golden record", using extract, transform and load (ETL) processes before importing the data.

1.3 MOTIVATION

MDM(master data management) is a set of Disciplines that provide a consistent understanding of master data entities, and their relationships. It is a set of Technologies that provide a mechanism for consistent use of master data across the organization, prescribed by governance policies. MDM is an essential product of IBM.

Infosphere Information Server provides a single unified platform that enables companies to understand, cleanse, transform, and deliver trustworthy and context-rich information.

A connector is required from the Infosphere information server for the MDM Database as there is no such connector in the information server till date. As a part of Information server, I got the responsibility of providing this connector from information server to the MDM. This connector is being designed to achieve connectivity between the IBM Infosphere Information server and IBM infosphere Master Data Management.

This connector is the java based connector, developed in java language. This is developed from the scratch and provides the basic functionalities required from any connector used in data stage. The real motivation behind developing this particular connector or stage is to integrate MDM with the information server which will increase the efficiency and productivity of MDM and enhance the functionality of Information Server as new connector or stage is being added into information server. There are two types of customer for MDM:one that extracts data and do little transformation on it, second that extract data and do complex tansformations on it. This connector is being designed for the second type of customers.

Any Enterprises which wants to accomplish its goals should always follow data governance rules. Everyone has different approaches according to their company needs and industry role. To make this possible companies needs to recognize and make use of already existing resources within and make most of it. Hence there is still a lot of distance between this goal and business process.

Hence MDM plays a vital role in todays rapidly growing technologies, any company can achieve their long end goals if they have mature data governance policies which MDM provides, across the company's customers and products.

The level of maturity and MDM determines the amount of impact of poor informatin management and data breaches. It also recquires the together working of people and the prosee and technology.

Some of the key points are :

Master data is usually spread out through the organisation with its sub organisation, take overs, alliances, acquires so on, resulting in dis ordered and redundant data which will effect bussiness process.

An exact interpretation is the key to business and results into most valuable resource

such as product information, customer information, billing information. And data keeps changing frequently, gets updated, its very difficult to maintain the single variant of correct information from all the sources.

MDM focuses on risk management and manages multiple domains of information, and include improved cost benefits and standards

1.4 PROBLEM STATEMENT

The project is defined to develop an MDM platform and MDM Input/Output stage so that MDM can be seamlessly integrated into the Information Server ETL job flow.

Java management extension (JMX) technology is been used to remotely manage the MDM jobs and that needs to be done at Infosphere Information server and which will convert the data into the form as required by the MDM automatically and pass that data to MDM for processing. In this way connectivity is achieved between Infosphere Information Server and Infosphere MDM.

Different sources defined are data input sources for the information, using any extract transform and deliver tool such as information server the information is pre processed. Then the data is input to the various steps defined by the MDM process.

To the point of view of professionals, it helps the organization to figure and assess their data model and flow of information and the way it is stored and used. And to the academics it is using of different knowledge of probability, hashing, bucketing applied knowledge. The organizations should be made flexible to undergo various changes such as technical, managerial, and operational during the process.

1.5 OBJECTIVE

From a corporate point of view, the objective is to give organizations the possibility to asses their own MDM maturity and benchmark against other organizations. This helps the firm figuring out improvement areas to become more efficient. From an academic point of view, the objective is to contribute to the body of knowledge on the field of master data management maturity assessment because little research was conducted on this particular topic so far yet it has a remarkable practical relevance. Some companies simply pile up masses of data with the expectation of gaining benefits from this. This expectation will not come true because the pure existence of data will not lead to any virtues. Even worse, it gets more and more complicated to find the particular relevant piece of data if it is somewhere among huge amounts of unmaintained data. The pure possession of data will not lead to anything if there is no logical structure that makes it possible to mine data according to criteria. Data only describes facts, there is still the lack of a judging ,interpreting or action triggering dimension . Problems with MDM can occur due to its complexity.A responsible person is confronted with a highly complex topic on the one hand and on the other hand, he is also overwhelmed with the saturated offerings of software solutions regarding this topic.The organization has to be prepared for the technical, operational and managerial changes that will occur throughout the process of MDM.Master data management is often not a sovereign management task, but just a sub area of business units. There is one person responsible for master data in his scope, but that person is not informed about data in other processes, even though the data might be the same and even changed in other processes.

The objective of the project is to develop an MDM Input/Output stage so that MDM can be seamlessly integrated into the IBM InfoSphere Information Server ETL job flow. This integration is achieved by automating couple of steps performed by the MDM.These steps need to be done at Infosphere Information server and it should convert the data into the form as required by the MDM automatically and pass that data to MDM for processing. The In this way connectivity is achieved between Infosphere Information Server and Infosphere MDM.

1.6 SCOPE

MDM has three versions InfoSphere Master Data Management Advanced Edition: Strategically transforms your organization through improved business processes and applications.

Standard Edition: Delivers business value for MDM projects with the quickest time-to-value. Collaborative Edition: Streamlines workflow activities across users who are involved in authoring and defining master information.

The different areas of master data may be in the context of customer data, product information, transactional data. Accounting and financial information is also important and used across the organistion. There can also be non-transactional data having the refernce information.

In the MDM platform resolutions and standards are enforced thought the enterprise. Various services are present to manage the master data. Successful MDM requires proper administration and collaboration between technology and bussiness.

Data from the various servers are processed by the Infosphere Information Server and convert in the format required by the MDM. The export and Import of the data as required by the different editions is separately processed for each edition of MDM. There is some manual steps which is currently performed at MDM side by the user in the MDM workbench provided in the form of user interface.

There are interfaces to achieve the connectivity between Infosphere Information Server and MDM but there is no mechanism to make them work together till date. With the help of this project we are developing an input output stage so that MDM is integrated into Information Server ETL job flow. Some of the jobs that are performed at MDM side is now moved to Information server so that efficiency of the overall system or organization can be increased. MDM Connector is capable of performing many functions like initial load,memget and memput to the MDM Server.

1.7 METHODOLOGY

Complete master data management solutions require the entire lifecycle of the master entities to be managed from within the master data management solution. Controlling the entry of the master data allows the enterprise application to proactively manage the quality of the data. Although an enterprise implementation will be both the system of entry and system of record for all master data entities, it may still require mapping data to other applications.

It is not realistic for an organization to get all of their systems to use the exact same set of data. Some transformations will still be required to run the process systems. This does not mean that every defining characteristic of an entity is managed within the solution. Those defining attributes unique to an organization's system operation should be managed within the source system where they have relevance. The enterprise solution should provide a broad range of entry points to be a viable option as the system of entry. These are the key processes for any MDM system.Profile the master data. Understand all possible sources and the current state of data quality in resource. Consolidate the master data into a central repository and link it to all participating applications Govern the master data. Clean it up, duplicate it, and enrich it with information from 3rd party systems. Manage it according to business rules. Share it. Synchronize the central master data with enterprise business processes and the connected applications. Insure that data stays in sync across the IT landscape. Leverage the fact that a single version of the truth exists for all master data objects by supporting business intelligence systems and reporting.

Existing System:

MDM has three versions InfoSphere Master Data Management Advanced Edition Strategically transforms your organization through improved business processes and applications.

Standard Edition: Delivers business value for MDM projects with the quickest time-to-value. Collaborative Edition: Streamlines workflow activities across users who are involved in authoring and defining master information.

Data from the various servers are processed by the Infosphere Information Server and convert in the format recquired by the MDM. The export and Import of the data as required by the different editions is separately processed for each edition of MDM. There is some manual steps which is currently performed at MDM side in the form of a graphical user interface.

Proposed System:

A system that has a single export import utility for all the three editions and use the MPINET server in the data engine which provides TCP/IP socket connections for the various API calls from the clients.

The main purpose of this system is to achieve connectivity between IBM Infosphere information server and IBM Infosphere MDM server.

IBM INFOSPHERE MDM

IBM INFOSPHERE INFORMATION SERVER

This connectivity is achieved by developing an Input Output stage betweem MDM and Information Server.

In the proposed system it is required to Provide a native Connector from Infosphere Information Server (DataStage )to InfoSphere MDM. This will be known as MDM input/output stage.

MDM Input/output stage is developed such that it should have following functionalities:

INITIAL LOAD

MEMGET

MEMPUT

INITIAL LOAD:

In Initial load data is initially loaded into MDM Server. For example any organization is using MDM, initially there is no data in MDM server engine, Initial load is the process of loading data into MDM. This will be the output of the MDM Connector and input for the MDM engine for a particular organization. This MDM Connector is provided by the Infosphere DataStage.

There are various jobs inside Initial Load: One of them is

MPXDATA-

In MPXDATA, the input to the MDM Connector is the input file, input file contains the input data. In MDM data is stored according to a data model, there is a different data model depending upon the member type. This data model is the metadata of MDM for an organization. This metadata is derived from the MDM using various api’s and used in the MDM Connector to define the mappings between the source data and the actual data needs to be stored in the MDM.

The MDM connector have a GUI interface which will have a same functionality as MDM workbench and thereby avoiding manual steps from end user perspective. This GUI interface is required to create a configuration file manually as required by the user.

The output of the MDM Connector is the configuration file and mpxdata.xml file which is used as an input by the JMX Client. By using the mpxdata.xml file JMX Client is creating the different segments in the MDM Server and loading the data in those segments accordingly.

MEMGET:

MEMGET is a type of interaction with the MDM engine to extract data based on key values. Fetches member records from the MDM database using a key such as : Enterprise ID, Source code and Member ID number or Member Record number. Output from the Master Data Engine a MemRow List, which is again a collection of rows that represent one or more member objects. You can retrieve members by executing the MemGet Interaction, which also takes a MemRow List as a parameter.

The MEMGET functionality needs to be performed by the MDM Connector. In MEMGET , user will extract some data from MDM Server engine, depending upon the key values. The user can extract data by giving some filtering conditions. The MDM Connector will provide a GUI interface to select the filtering properties for the MEMGET. The output of the MEMGET will be the data desired by the user, In MDM Connector some mapping will be performed to extract user desired data depending upon the fields specified by the user.

The MEMGET functionality is performed by the MDM Connector in output context. The result of the MEMGET functionality is input for the MDM Connector and output from the MDM engine. For MEMGET functionality MDM Connector will act as MDM Input Stage. In MDM Connector there is a graphical user interface for MEMGET which is designed in java swings and invoked from MDM Connector using DMDI(Dynamic Meta Data Import Interface) . This GUI is designed to take values of various filtering properties required for MEMGET operation by the user.

MEMPUT

MEMPUT is a type of interaction with the MDM engine in which data is inserted into the MDM database.The Input parameters is constructed through segments like MemHead and MemName. Each one of these segments represents a row. Each row contains different attributes that are used to make up a member. Each row is then added to a MemRowList and passed in as a parameter to the MemPut Interaction. TheMemPut interaction sends these attributes to the Master Data Engine where the rows are processed and relevant tables in the Master Data Engine database are either inserted or updated.

This MEMPUT functionality should be performed by the MDM Connector. The MEMPUT is basically an input to the MDM database and output from the MDM Connector. For example if the user wants to enter data into the existing MDM database then MEMPUT can be used. MEMPUT functionality is performed by the MDM Connector in the output context.

For MEMPUT functionality , MDM Connector will act as MDM Output Stage.