Big Data With Cloud Computing In Organizations

Published Date: 02 Nov 2017

Introduction

Today, in any large city of this world, it is not possible to go unnoticed, without a CCTV camera picking up your movement and storing these images for the future. Advancements in technologies in the area of DNA, fingerprinting, eye scanning etc. are such that crimes committed over 30 years ago are being solved as technology provides the means to identify the previously anonymous.

Sadly, the same cannot entirely be said in organizations & social life. There are many obstacles, including culture, language, and general common practice such as Data Protection Acts. Yet the crimes and offences committed against organizations and individuals such as data theft, sharing of personal information etc. seem to frequently remain undetected simply because we can’t find a way to share those nuggets of data.

According to an IDC Digital Universe Study, social network offered by Facebook. Here's an overview of data passes through the Facebook server:

Currently, 1.8 zeta bytes of data are being created on Facebook every year.  Every day, 2 million blogs are being posted, 172 million users visit Facebook, 51 million minutes of video are uploaded, and 250 million digital photos are being shared.   

We are doubling the world’s information every 18 months; however, information with Facebook is doubling every 12 months.

Variety of services being offered by Facebook is exponentially growing. For example, plain text messages, photo sharing, live video streaming, online marketing, time line etc.

Issues related to privacy of personal data, restriction on sensitive data, blockage of confidential data, respect for values and culture of a part of the society are becoming more critical.

There are many circumstances where we got the answers. They just stored in different workstations without the technology to collaborate, compare, and map.

Collaboration is the answer. That’s certain. And as Cloud Computing technology goes through a radical transformation, the time is right to design the solution which improves our organizations & social service output. But what do we mean by collaboration and what do we expect it to deliver in this context?

Big Data

The term Big Data is about growing challenge of the new generation technology which is designed by concerning economically low investment and can extract large volume of a huge variety of data capturing, analysis and processing. The definition of "Big Data" varies with the situations. Big Data is a mixing of structured, semi-structured and unstructured data that breaks the barriers of traditional database.

McKinsey and Company defines the Big Data as

"Datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze"

According to the Teradata magazine article, the Big data can be defined as

"Big data exceeds the reach of commonly used hardware environments and software tools to capture, control, manage and process it within a tolerable elapsed time for its user population"

According to the Jon Kleinberg, a computer scientist at Cornell University , the Big data is itself vague but something is real in it. It can be defined as

"Big Data is a tagline for a process that has the potential to transform everything. It is really about new uses and new insights, not so much the data itself."

The scope of big data can be understood by four V’s of Big Dat:a, which are explained as below.

Four V’s of Big Data

Volume: The benefit gained from the ability to process large volume of information is the main attraction of big data analytics. Apart from Facebook statistics mentioned above, we continue to generate 294 billion emails every day, in which many consider emails are an outdated form of communication.

Velocity: Data Velocity is the speed at which data is growing and this extreme speed is taxing our current information technology capabilities. According to an IDC Digital Universe Study, we are doubling the world’s information every 18 months. [12]

Variety: The Big Data can be categorized into structured, semi structured and unstructured form.

The structured form is the most traditional way of storing the data. Financial transactions including movie ticket sales, online bill payment, restaurant sales, etc., are generally structured and it impacts in a little fraction of the data running around the global networks today. Unstructured data is a primary source of growth in variety of video or audio data.  In a day nearly 19 million hours of music is uploaded or downloaded in the free music service. There are over 864,000 hours of video uploaded to YouTube each day. The semi-structured big data can be accomplishing from many sources as text files, XML files etc.

Validity is a singular term designed to characterize the quality, extraction, derivation, value, reliability, setting, and context for the data. Structured and unstructured data requires valid data from the trusted sources and it should follow the data from procurement to retirement because the trusted sources are highly valued than data from a new or casual sources. Over time, the new source may be further tested and validity of prior data from that source may increase or decrease.

Cloud Computing

In cloud computing, the word cloud is used as a metaphor for "the Internet," so the phrase cloud computing means "a type of Internet-based computing", where different services -- such as servers, storage and applications -- are delivered to an organization's computers and devices through the Internet. [15]

The cloud has become a viable, mainstream solution for on demand data processing, storage and distribution. It improves not only the speed, but also the quantity and quality of resources available in an organization.

Cloud computing permits the consumers and end-users to use those applications which are in the cloud without installation and can be access their personal files from any computer by using only internet in high speed. This technology allows for much more efficient computing by centralizing data storage, processing and bandwidth. A simple example of cloud computing is Yahoo email, Gmail, or Hotmail etc. All you need is just an internet connection and you can start sending emails. The server and email management software is all on the cloud and is totally managed by the cloud service provider Yahoo, Google etc.(fig 1)

Figure 1: Services of Cloud Computing

Software as a Service (SaaS): In the SaaS model, the application software are install and operates by the cloud provider and the cloud users can access these application directly on the cloud without installation. The users do not need to accomplish the cloud infrastructure and platform on which the applications are running. SaaS eradicates the need to install and run the application on the cloud user's own computers simplifying maintenance and support. Examples are Google Apps, Games, and Facebook etc.

Platform as a Service (PaaS): In the PaaS model, the cloud providers provide a computing platform that includes programming language execution platform, operating system, database and web server. The developers can develop and execute their applications and get the result on a cloud platform without the cost and complexity of procuring and managing the underlying hardware and software layers. Examples are: Google App Engine, Windows Azure Compute.

Infrastructure as a Service: IaaS providers provide workstation, as physical or as virtual machines, data storage devices, load balancer and many more other services. These clouds offer some more resources such as images in a virtual machine, image library and file-based storage, firewalls, IP addresses, virtual local area networks (VLANs), and software bundles. IaaS cloud providers supply these resources on demand from their large pools installed in data centers. Examples are:  Google Compute Engine, HP Cloud.

Knowledge as a Service: It is a new service that the cloud with provide in near future between platform as a service and software as a service. The Service Cloud provides the tools that need to manage knowledge at enterprise scale. But it also delivers the same great ease of use that salesforce.com is known for. That means you’ll benefit no matter what size or how complex your business is.

Collaboration of big data with cloud computing

Traditionally, the huge amount of data stored in the systems of large organizations like IRCTC etc. When required, it becomes very difficult to collaborate, match and map the data by applying the queries on such large databases.

Therefore, we find slow and poor performance of our existing systems. At one side, system takes a lot of time to respond to our queries. On the other hand, system remains idle for most of the time when not needed.

Here, collaboration of Big Data with Cloud Computing technology can transform the performance of the system drastically. Through this collaboration, queries are run on a virtual platform of enormous capabilities and such platform can be used by different users at different times. This way, resources are utilized optimally and cost efficiently.

Figure 2: Big Data – Cloud Computing Collaboration

The people invested a huge amount in social services, security, health and child education related processes. The different societies such as schools, police stations, hospitals or doctor clinics, all have information about children and families, their education status, attendances, appearances, offences, and illnesses and injuries. But they are fragmented. There is a big challenge of collaboration to relate one to the other society. Making it happen in real time and providing cross analysis of data to deliver intelligent decision making ability is the challenge of making a difference.

The State and local governments are moving around the Cloud computing. Big Data has the competency to analyze vast and growing data volumes and get fast answers for tough queries. And case and content management, at the center of application service architecture, means we have the means to collaborate these answers and support the various multi-disciplinary teams that make up our social, security, health, education and human services.

However, companies that work with big data have been unable to realize its full potential due to the significant challenge of moving large volumes of data in, out and across remote cloud infrastructure. Cloud computing vendors offer big-data tools that, paired with the ability to rent scalable on-demand compute and storage resources, can provide a potent on-paper justification for analyzing data in the cloud.

The Big Data architecture introduces a fourth layer in the cloud computing stack.  Knowledge as a Service joins the traditional cloud layers (Infrastructure as a Service, Platform as a Service, and Software as a Service) as a focused layer dedicated to the management and analytics of Big Data including binding concepts such as pedigree, lineage, and provenance of data. 

Impact of Collaboration:

The collaboration between Big Data & Cloud computing have impact our lives in the following manner:

Generation of massive volumes of data "Big Data" through online social networking, e-commerce purchases, chatting etc. can be used to create personal & behavioral profiles of people. This can be used to store, manage and analyze on demand for certain objectives.

Useful scientific historical data based on our experiences and observations related to astronomy, genomics, medicine, zoology, and even environment, weather and transportation data can be used more efficiently with the cloud computing.

We also have the Humanities and Social Sciences with its vast resources on historical documents and social interactions data. And then we have Business & Commerce, Entertainment, and social networking which would possibly be the current leader in data growth on a daily basis. This can be managed with the help of Cloud Computing.

This collaboration can also impacts IT employment:

New categories of jobs arising from cloud computing. "Skills now in demand include private cloud developers and administrators, departmental liaisons, integration specialists, cloud architects, and compliance specialists."

Cloud is still a fraction of enterprise IT spending.

The small pool of experienced cloud talent will pose a challenge to employers and opportunity to IT pros

Surveys show few companies adopting cloud for the purpose of reducing IT headcount.

Cloud is a growing sector of the economy, creating jobs along with it.

As cloud providers grow and grow and grow, they're going to need lots of IT staff to maintain all those servers.

Cloud or no cloud, the number of job postings in the cloud computing industry is growing so rapidly that there aren't enough qualified workers available.

Businesses that embrace the cloud will save big.

Best practices across different industries can be replicated easily. Which could be:

Light Weight Services

Mobile First User Interface – Can be accessed through a tablet or browser based desktops

Fast Data

Application level management

Elastic scaling – resources can scale up with the demand

Self-healing – Unusual crashing of system components can be avoided / recovered easily with the help of Cloud Computing

Future of Collaboration

In future, the cloud computing will simply provide the best platform for the storage, management, process and subsequent distribution of Big-data. It will permit a lot of cross-referencing with distributed databases and also permit scalable analytic innovations because of the accessibility provided by cloud computing. By allowing data scientists to connect to various databases though cloud collaboration, they can make use of distributed computing for faster processing and analysis of the data which will allow for almost instantaneous results.

In future, developers will realize that development in the cloud is not that different. Enterprise applications will be approved on public clouds, which would mean that businesses opportunities will be more in the social world. Back-up & disaster recovery for clouds will also be in place. Businesses will accept clouds in a more practical way both in terms of cost as well as usage. In telecommunication the cloud computing will also provides the services to the customers such as playing the games without installation on the mobile means saving the space.

Conclusion

Through this paper, we have learnt that "Cloud Computing" has the potential to develop application specifically for large databases like "Big Data" through analytical innovations.

Being the social and financial importance, extraction of relevant information from "Big Data" would be one of the most challenges in future. It will extensively be used to design government public service policies as well as designing marketing policies.

The application of "Cloud Computing" will not only be limited to provide software for data extraction and knowledge management but also be extended towards providing infrastructure & platform related services.