Data Warehouse As A Data Mining Source

Published Date: 02 Nov 2017

"Business intelligence (BI) is a set of methodologies, processes, architectures, and technologies" (Wikipedia, the free encyclopedia, n.d.) that find useful and meaningful "information" from raw "data". With BI an organization can handle large amounts of information to help to identify and develop new opportunities and also decide for effective strategies which could help to provide competitive market advantage and stability.

In 1970s Business Intelligence derived from decision-making support technology. Later it experienced a complex and gradual evolution including Transaction Processing System (TPS), Executive Information System (EIS), Management Information System (MIS), Decision Support System (DSS) and other stages. The Gartner group in 1996 defined BI as series of systems which has data warehouses, data analysis, data mining and which help the organization to make a better decision and keeps its leading position in the competitive market. Business intelligence is using information of company’s past performance to predict the company’s future performance. Emerging trends from which the company might profit could be revealed by BI.

Using BI technologies, you can provide historical, current and predictive views of business operations. Some of the common functions of BI technologies could be named as reporting, online analytical processing, analytics, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics and prescriptive analytics.

One of the main and important goals of business intelligence deployment is to support making better business decisions. So in other words a BI system can be called a decision support system (DSS).

Knowledge management

Knowledge Management (KM) is a concept and a term that get up around two decades ago, approximately in 1990. Very early on in the KM movement, Davenport (1994) offered the still widely quoted definition:

"Knowledge management is the process of capturing, distributing, and effectively using knowledge."

The above definition has the advantage of being simple, stark, and to the point. Few years later, another second definition of KM created by the Gartner Group which is perhaps the most frequently cited one (Duhon, 1998):

"Knowledge management is a discipline that promotes an integrated approach to identifying, capturing, evaluating, retrieving, and sharing all of an enterprise's information assets. These assets may include databases, documents, policies, procedures, and previously un-captured expertise and experience in individual workers."

Both definitions are very corporate orientation and very organizational. KM, historically at least, is primarily about managing the knowledge of and in organizations.

Data Warehouse

…content for Introduction…

Data Warehouse as a Data Mining Source

An excellent source of data to locate and mine is an enterprise data warehouse. "Because of the nature of a data warehouse, most pertinent data that has been selected by analysts and business users should be located within the warehouse structure. In addition, this data is organized and stored for the explicit purpose of reporting. Through the data warehouse, further processing of OLAP data can occur. This processing can take the form of additional aggregations into multidimensional cubes (i.e., SQL Server 2000 Analysis Services Cubes) or undergo further segregation into organizational data marts.

The data mining process will utilize the data in the enterprise data warehouse, based on user selection and location of pertinent data, to test and validate a data mining model. It is important that the data be granular enough to analyze. Data that is characterized by significant aggregations beyond the original grain of the data will not produce significant results when used to create or test against a mining model.

An enterprise data warehouse is a prime source for data mining data because the data housed within the warehouse has already undergone significant data additions, modifications and cleansing based on business rules and processes. Refined Extraction Transformation and Loading (ETL) processes are required for reliable OLAP and enterprise data warehouse reporting. It is the ETL process which is responsible for cleansing bad data from the OLTP source, reclassifying or aggregating granular transactions from the operational system, and enriching the data with more readable and comprehensible data as opposed to the operational codes and abbreviations used in an OLTP system. Once the data has been sufficiently cleansed and refined, it is ripe for data mining" (Chaterjee, n.d.).

Typical data warehousing implementations in organizations will allow users to ask and answer questions such as "How many sales were made, by territory, by sales person between the months of May and June in 1999?" Data mining will allow business decision makers to ask and answer questions, such as "Who is my core customer that purchases a particular product we sell?" or "Geographically, how well would a line of products sell in a particular region and who would purchase them, given the sale of similar products in that region?"

Data Mining

In today’s world, every business, company and organization has its own large amount of data. They usually use their own data for their future decisions, research and their development. The data in their databases is on their hand when they require it. But the most important thing is to analyze the data and find important information. If you want to grow rapidly you must take quick and accurate decisions to grab timely available opportunities (Arthur, n.d.).

Data mining allows users to sift the data in data warehouses and get enormous amount of information. With this process you can access the business intelligence gems. Using the process of data mining, you can extract required valuable information from data. So data mining is about refining data and extracting important information. Data mining is the process of extracting hidden knowledge from large volumes of raw data, it can also be defined as the process of extracting hidden predictive information from large databases (Chaterjee, n.d.).

The process of data mining is mainly divided into 3 steps;

Pre-processing

It is about collecting large amount of relevant data

Mining

It is about data classification, clustering, error correction and linking information

Validation

It is about trust on new information

Data mining should not be considered as an intelligence tool or framework. Business intelligence, typically drawn from an enterprise data warehouse, is used to analyze and uncover information about past performance on an aggregate level. Data warehousing and business intelligence provide a method for users to anticipate future trends from analyzing past patterns in organizational data. Data mining is more intuitive, allowing for increased insight beyond data warehousing. An implementation of data mining in an organization will serve as a guide to uncovering inherent trends and tendencies in historical information. It will also allow for statistical predictions, groupings and classifications of data.

A massive quantities of data is collected, refined and comprehended by companies or organizations. Using data mining techniques on existing software and hardware platforms, make it possible to rapidly to improve the value of existing information resources. Also data mining techniques can be integrated with new products and systems as they become part of the system. They can analyze massive databases to deliver answers to many different types of predictive questions when they are implemented on high performance client/server or parallel processing computers.

One of the advantages of data mining tools is that they predict future trends and behaviors, this help the business to make proactive, knowledge-driven decisions. Using data mining tools you can answer business questions that traditionally were too time-consuming to resolve. Data mining is, in some ways, an extension of statistics, with a few artificial intelligence and machine learning twists thrown in. Like statistics, data mining is not a business solution, it is just a technology.

Benefits of Data Mining for Organizations

Fast and Feasible Decisions

If you want to search for information from huge amount of data, it requires lots of time. It also irritates the person who is doing such. Not only when a person is doing such work the possibility of making mistakes and incorrect decision increases, but also with annoyed mind no one can make accurate decisions for sure. By help of data mining, you can easily get information and make fast and authentic decisions. It also helps to compare information with various factors so the decisions become more reliable.

Powerful Strategies: with the information which is available after the data mining, you can make different strategies. In other word by analyzing information in various dimensions you can make different strategies and implement them. This could help the organization to effectively expand its business boundaries and making authentic decisions.

Competitive Advantage: with the information in your hand you should try to compare it in different aspects and doing competitive analysis and making corrective decisions. This will enable the company to gain competitive advantage.

When did data mining begin?

Data mining techniques are the result of a long process of research and product development. This evolution began when on computers business data was first stored, then there were improvements in data access, and more recently, there are new technologies that allow users to navigate through their data in real time. Data mining takes this evolutionary process beyond retrospective data access and navigation to prospective and proactive information delivery. Data mining is ready for application in the business community because it is supported by three technologies that are now sufficiently mature:

Massive data collection

Powerful multiprocessor computers

Data mining algorithms

The core components of data mining technology have been under development for decades, in research areas such as statistics, artificial intelligence, and machine learning. Today, the maturity of these techniques, coupled with high-performance relational database engines and broad data integration efforts, make these technologies practical for current data warehouse environments.

Data mining techniques can yield the benefits of automation on existing software and hardware platforms, and can be implemented on new systems as existing platforms are upgraded and new products developed. When data mining tools are implemented on high performance parallel processing systems, they can analyze massive databases in minutes. Faster processing means that users can automatically experiment with more models to understand complex data. High speed makes it practical for users to analyze huge quantities of data. Larger databases, in turn, yield improved predictions.

How does data mining work?

The technique that is used to perform these feats in data mining is called modeling. Modeling is simply the act of building a model in one situation where you know the answer and then applying it to another situation where you don’t. For instance, if you were looking for a sunken Spanish galleon on the high seas the first thing you might do is research the times when Spanish treasure had been found by others in the past. You might note that these ships often tend to be found off the coast of Bermuda and that there are certain characteristics to the ocean currents, and certain routes that have likely been taken by the ship captains in that era. You note these similarities and build a model that includes the characteristics that are common to the locations of these sunken treasures. With these models in hand you sail off looking for treasure where your model indicates it most likely might be given a similar situation in the past. Hopefully, if you have made a good model, you find your treasure.

The process of creating the data mining model is directly dependent on the methodology used to feed the entire data mining process. In essence, the method used to make data available to be mined governs the process used to create the data model. If a solutions architect designed a specialized OLAP data cube in Analysis Services to serve as the primary source of data mining data, then an OLAP data mining model would be created, as opposed to a relational data mining model.

This act of model building is thus something that people have been doing for a long time, certainly before the advent of computers or data mining technology. What happens on computers, however, is not much different from the way people build models. Computers are loaded up with lots of information about a variety of situations where an answer is known, and then the data mining software on the computer must run through that data and distill the characteristics of the data that should go into the model. Once the model is built it can then be used in similar situations where you don’t know the answer.

In what areas is data mining profitable?

Data mining has been deployed by a wide range of companies successfully. This technology is applicable to any company looking to leverage a large data warehouse to better manage their customer relationships. early adopters of this technology have tended to be in information-intensive industries such as financial services and direct mail marketing,

Two critical factors for success with data mining are:

A large, well-integrated data warehouse

A well-defined understanding of the business process within which data mining is to be applied (such as customer prospecting, retention, campaign management, and so on)

Some successful application areas include:

Pharmaceutical companies

Credit card companies

Transportation companies

Large consumer package goods companies (to improve the sales process to retailers)

Each of these examples has clear common ground. They leverage the knowledge about customers implicit in a data warehouse to reduce costs and improve the value of customer relationships. These organizations can now focus their efforts on the most important (profitable) customers and prospects, and design targeted marketing strategies to best reach them.