Introduction Of Software Testing

Published Date: 02 Nov 2017

Software testing is an analysis performs to provided stakeholder or relevant parties with information about the qualities of the deliverable. The output of the process is use: -

To determine that software are satisfying the qualities aspect

To ensure that product are fit to be use

To discover software defect or failure to avoid loss of goodwill and foreseen financial loss.

Traditionally, testing is done manually without any aid to tools and relies heavily on human resources and tends prompting for human error. This might happen because lacking of knowledge in selecting and implementing the suitable tools. Besides that, tools such as QACentre are come with a price tag. This given opportunity for software enthusiasm from around the globe teaming up ethically to enhance the process and to develop component or software which access publicly by everyone and the component which might consist of test automation component, bug management, object component or others.

This project is taking opportunity to making use of the open source component to design and develop a Web Automation Tool (WAT) that could ease software tester to perform testing, and it could capable of maximizing the resource utilization.

Statement of the Problem

In 2012, Monetary of Singapore (MAS, 2010) takes supervisory action against DBS Bank for the service outage for it online and branch banking system which causes inconvenient to the bank customer which translate Bank are required to set aside S$230million in regulatory capital on improving the Bank infrastructure. As of now, none of this happens in Malaysia yet and indirectly this losing customer confident and goodwill to the Bank itself.

Electronics Banking is Malaysia is very competitive and according to Bank Negara Malaysia (BNM, 2013) statistics in Year 2012, we have total of 15 trillion transaction performed online which included consumer and corporate for different type of various transaction type across all Banks. Due to pressure of competitiveness of the environment, Bank is pushing for new product and offer to be available via online because consumer able to reach for the services anytime anywhere.

Within the Bank itself, development and implementation might not be as easy as it look. New feature and function introduce into inline banking required huge amount of regression on existing function to make sure that implemented function does not break or cause further issue.

Testing done manually might not be able to cover all the scenario and scope of the application. With automation tool, scenario can be recorded and program for further usage and repetitive testing will be able to execute scenario.

Objective

The objective of the project is as below: -

Firstly, investigation is required to research understand what are the open source component that are with such capabilities. Besides that, a decision making is required to finalize which component will be used to develop the Web Automation Tool.

Secondly, is to highlight strength and weaknesses of GUI automation tools.

Thirdly, to build GUI using the open source component to perform web automation testing on web application with proprietary function build to support EASY Internet Banking [ [1] ] application. Automation testing only cover server site validation and client site checking will be enhanced in future development.

Fourthly, projects are required to clearly identify possible area of improvement that can use for future enhancement to the system.

Road map of Report

This dissertation consist of chapters and following is the outline of the content of each chapters

Chapter One: Introduction

The project introduced definition, overall purposes, objective and aim of the project.

Chapter Two: Literature Review

This chapters enclosed the literature review where is given an overview of open source and brief history of software testing

Chapter Three: Analysis

This chapter describes the analysis of the automation component selection, it will also analyse in-term of logical data model.

Chapter Four: Design

This chapter describe design process together with database and interface design.

Chapter Five: Development and Implementation

This chapter provided detail for development and implementation.

Chapter Six: Testing

This chapter report the detail testing of the automation tool by using an example, together also briefly explain the test case, result obtained with screen capture of the process taken.

Chapter Seven: Summaries and Conclusion

This chapter will summaries and conclude overall of the report in term of objective archived and finally highlight any opportunity for future enhancement to make it better.

Technology Overview

To development the GUI automation, several tools and technologies will be used in this project. Below is the brief introduction of their usage.

Hardware and Operating System

Below is the development platform which uses to produce the GUI automation.

Microsoft Windows 7

Compaq 511 laptop computer (Intel® Core™2 Duo CPU Processor 2GHz)

RAM 4GB (3.00GBUsable)

Internet access with minimum of 1Mbps

Database Tools

For data storage below SQL database been selected.

MS SQL 2005 Standard Edition

Application Development Tools

For development IDE, below is the selected.

Microsoft Visual Studio 2010 Professional Edition

.NET Framework 4.0

Open Source Component (Selection will be finalize in section 4.1.2.5)

Chapter Two: Literature Review

Introduction of Software Testing

Software testing is a process of development cycle regardless of which development methodologies is preferred. The purposes of testing are to discover error and to rectify that it satisfied the specified requirement. During the development life cycle, software engineer and tester are working together to find bugs and ensuring product must include all function and be compatible with the targeted environment.

Traditionally, all testing are to be done manually but since the birth of software industry. Software engineer has made great effort to improve the environment by automated the testing process. With the automated tools software testers are only required to records and verified the test cases for executed later.

History of Automation

According to Gerald (Gerald D.Everett & Raymond McLeod Jr, 2007), software testing began to mature around 1980. At the beginning, tools that are available are mainly design for single purposes which focus on functional testing. After gaining popularity, the tools emerge with extra feature such as test management tool. In year 2000, software provider such as CA, HP, IBM, Compuware and other has developed a comprehensive tool suite for commercial purposes and such tool will be able to provide test management, functional, security testing all-in-one.

Benefit of Web Automation

Competitiveness pushing business desperately to provided more feature and function. It indirectly, increase the revenue and ease of transacting banking transaction as compare to branches, while late produce might loss revenue and market shared.

Pressure not only limited to competitiveness, it also pushing delivery team to streamline the resource utilization and cost as well leading organization to sourcing opportunity to adopting automated tools as part of their processes.

Automating testing can provide few key benefits as below: -

Test coverage

Complexity of application growth as more and more feature introduces and provide to customer, and more resources is required on testing to have better test coverage.

Imagine, a feature required modification of 10% on existing code changed. However, the software testers are required to design test case more than that, in this way manual test will never able to catch up with the test cases without investment on more resources.

In worst scenario, when time is the constraint software tester often reducing the test coverage with assumption that "Other should work as normal", this post a greater risk to existing feature that might break the compatibility.

Figure 1: Feature vs. Time

Automation should be able to assist software tester to accumulate test cases into repository over period of time so that both existing and new.

Humanly impossible

Executing performance testing using a manual approaches is a resources intensive activity. Considering the cases of simulation a function or workload to be tested in the range of 100-1000 active customer transact ion without automation. It is possibility by request or hiring extra resource to complete the task. However, how efficient or effective it is to re-executed the test cases again? Theoretically, any performance testing should be done with the assistant of a tool.

Increase market share

As mention is earlier section, Bank industry is a very competitive environment and as for electronics channel, it is a tool or weapon for Bank to set a higher standard and goodwill. Automation can assist to reduce the testing cycle by allowing it to be configuring by 24x7.

Reusability

Automation tools are design with purposes of ease software tester to reuse the test cases. When the test cases is getting mature and stable, software tester are able to reuse the repository to perform regression.

Cost saving

Last by at least is the cost saving. As illustrate in section 2.1.2.2, just imagine how much cost a project can be reduce by using automation.

With automation, software tester is able to set the tool to executed test case in a non-working hours or even weekend. Besides that, it can be also executed the test case in different machine by running different web browser.

Drawback of Web Automation

As we talking about the benefits of how much cost saving it can saved or how reusable it can achieve in previous section. Automation work when behaviours of the application are known to the tools, and when things don’t works as usual it is better to reprioritise the automation.

Nature of the application

Based on the research by IDC (David H. Deans, 2013), smartphone segment has the highest growth among desktop PC, portable and tablet. This proven in future, customer most likely will rely on smartphone to perform it daily needed.

Figure 2: Worldwide Smart Connected Device (2010-2012) (David H. Deans, 2013)

Based on the statistics, not surprisingly that more and more application will be targeting smartphone customer such as iPhone and Samsung because of portability, and it is a great opportunity for automation on such areas.

Figure 3: Worldwide Smartphone Market (OS Share) (David H. Deans, 2013)

As of now, automating on smartphone remain a very a very big question because of the behaviour of platform itself. As refer to Figure 3, Android is the most popular platform that chosen by manufacturer to boot their devices.

Version

Code name

Release date

API level

Distribution (March 4, 2013)

1.6

Donut

September 15, 2009

4

0.20%

2.0–2.1

Eclair

October 26, 2009

7

1.90%

2.2

Froyo

May 20, 2010

8

7.60%

2.3.3–2.3.7

Gingerbread

February 9, 2011

10

44%

2.3–2.3.2

Gingerbread

December 6, 2010

9

0.20%

3.1

Honeycomb

May 10, 2011

12

0.30%

3.2

Honeycomb

July 15, 2011

13

0.90%

4.0.x

Ice Cream Sandwich

December 16, 2011

15

28.60%

4.1.x

Jelly Bean

July 9, 2012

16

14.90%

4.2.x

Jelly Bean

November 13, 2012

17

1.60%

Figure 4: Android Version (Android Developer 2013)

In just a short period of less than 4 years, Android has release 10 different API level and each of the API have significant of improvement on the feature and hardware requirement. With such a rapid environment, automation on Mobile applicable is highly not suitable.

Automation tools are expensive

Automation tool price tag doesn’t seem to be cheap, it can be range starting from thousand for a standard base price without add-on such as: -

number of computers it going to install

number of software tester using the tools

or might be charge by using matrix such as logon session, test executed or

Ideally, an automated tool should be design to solved problems of the worlds, but automated tools are tightly couple to it function which cause single lead to single purposes tools. A functional testing tool is design to validation rules defined functional, and does not have the ability to execute task which mimic performance tools. These indirectly force project team to source for other suitable tools.

Boundaries of testing capability

Nature of the automation tool is to recording the event or behaviours of the transaction. When the certain improvement done on the specific module the previous recorded script might not be working because of the event of the transaction is changed.

Higher maintenance cost

Automation is design with purposes for saving cost and resources; this condition is valid only when test cases setup can be used to play over and over again. Cost is incur when test cases required revalidate and verified again due to changes on application.

Open Source

Open source is generally a concept and methodology that promote free distribution and gain access to a finishing product. The product itself is available to the general public for use and allowing anyone to use, modify and redistribution.

However, according to Open Source Initiative (Open Source Initiative, n.d.) that is some characteristic that that should be meet in following section.

Characteristic of Open Source

Free redistribution

No licensing shall be imposes on any party from selling or giving away the software as a component, and shall not require any loyalty or other fee for such a sales.

Source code

Program must include source code and must allow distribution in source code as well as in the form of binary or packages. The source code is prohibited to obfuscate or some form of encryption.

Derived works

The licensing must allow modification and redistribution under the same terms as the license of the original software.

Integrity of original source code

During compilation, source code can only allow to modify if the license allow the distribution of "patch files" with source code else it is required to carry a different name or version number from the original source code.

No discrimination against person or group or fields of endeavour

The license must allow it to be used by any individual person, group of people or profit and non-profit organization.

Distribution of license

The right of the redistribution is automatically inherited to the finishing product without the need for execution of an additional license by those parties.

License must not be specific to a product, other software or technology neutral

License shall not be restricting to a product, licensed software or organization.

Comparison between Open Source and Proprietary Software

"How much?" is often the most frequent question that will be prompt when purchasing a products and it is largely depends on the nature of the needed, and other factor such as complexity of products in term of integration and support.

However, when compare to proprietary source there are some behaviour that is better than open source such as documentation and systematic approach. It was mainly due to lack of responsibilities and focusing on fixing bug rather than documentation.

Figure 5, list the summaries of properties between proprietary and open source.

Property \ Type

Proprietary Source

Open Source

Copyright

Yes

No

Cost

Yes

No

Publish of Source Code

No

Yes

Free Software

No

Yes

Code Modification

No

Yes

Systematics Development

Yes

No

Frequent Release

No

Yes

Openness

No

Yes

Parallel Process

No

Yes

Knowledge Sharing

No

Yes

Private License

Yes

No

Profit by Organization

Yes

No

Figure 5: Properties of Proprietary and Open Source (Khanjani, 2011)

Type of Open Source

Chapter Three: System Development Methodology

Introduction

A system development methodology refers to the framework that is used to structure, plan and control the process of developing an information system. A wide variety of such frameworks have evolved over the years, and each with its own recognized strength and weaknesses.

One system development methodology might not necessary suitable for all types business and organization. Each of the available methodologies is best suited to specific kinds of projects, based on various technical, organization, projects and team considerations.

Waterfall Model

Figure 6: Waterfall Model

Principally, the projects are divided into sequential phrases with overlap and splash back acceptable between phrases. It is emphasis on tightening the control over throughout the rest of the cycles by using written documentation as well as through formal reviews and sign-off at the end of most phrases before beginning the next phrase.

Chapter Four: Analysis

Automation Tool Selection

Criteria for Automation Component Selection

As highlighted in section 1.5 (Technology Overview), Microsoft technology is chosen as the platform for the development of this project as compare to Linux or Mac OS. Below is the reason why such platform is chosen.

Ease of use

Microsoft platform and design is very user-friendly, where it using a single installation package of Windows Installer (MSI) or Executable (EXE) format. The installation process usually required user to click "Next" unless user chose "Advance" installation option.

Integration and Interoperability

As mention in section 1.5, Visual Studio been chosen. The installation step is straight forward together with a set of component available. As compare to it sibling "Eclipse", where setting up the development environment for Mobile and Java development is differently where separate component installation is needed. In separate event, Microsoft has the greatest backward compatibility support.

Documentation

Documentation is an important entity any aspect of the software development cycle, starting from initial requirement to implementation until maintenance. Lacking in documentation might slow down the research and development on the respective area and in order to verified it someone need to perform trial-and-error in making sure that it is really work compatible with existing functional, such trail-and-error is waste of resource if documentation is in place.

Another example, such as implementation documentation is design to assists and smoothen the deployment process, and avoid same mistaken to be happening again.

Analysis of Automation Component Selection

In the open source industry, there are numerous tools that are capable of providing the ability of doing automation. This section will analysis the capabilities of each tool and finalizes the component selection. The following section will provide sufficient information about individual tools

Web Application Testing in Ruby (Watir)

Watir (Watir, n.d.) is an open source (BSD) automation tool running on Ruby that supports Windows and Linux.

Below listed the environments that supported –

Operating System – Windows, Linux

Programing Language - Ruby

Browser – Chrome, Firefox, IE, Safari, Mobile Browser

Documentation – Blog from individual author

Web Application Testing in .NET (WatiN)

WatiN (WatiN, n.d.) is born on December 2005, after inspired by "Watir". It is aim with integrating web application testing possible for .NET languages.

Below listed the environments that supported -

Operating System – Windows

Programing Language - .Net

Browser – Firefox, IE

Documentation – Limited resource

SeleniumHQ

Originally, start in 2004 in ThoughtWork (SeleniumHQ, n.d.) in Chicago as "JavaScriptTestRunner" for testing internal application. The workgroup later expanded outside of ThoughtWork with researches located remotely across the worlds such as Japan and China.

WebDriver is a codename for one of its many components and is capable of supporting large number of browser vendor who have taken steps to make WebDriver a natives part of their browser. It is also the core technology in countless other browser automation tool (WebDriver, 2013), API and frameworks. This is because of WebDriver have a large number of contributor and good documentation.

Below listed the environment that supported -

Operating System – Windows, OS X, Linux, Solaris

Programing Language - .Net, Java, Python, Ruby, Perl, PHP

Browser – Chrome, Firefox, IE, Safari, Opera, Mobile Browser

Documentation – Blog from author and object reference (WebDriver, 2013)

HTTP Object

HTTP Request and Response (Connolly, 2004) is natively support by most languages such as Java, .NET, Ruby and others. The object is use for transporting request via tunnel of HTTP from client to server or via versa. Because it is leverage on the transport level, thus it is making the implementation harder and required skillset on transport level development.

Below listed the environments that supported -

Operating System – Windows

Programing Language - .Net, Java, Python, Ruby, Perl, PHP

Browser – N/A

Documentation – limited resource

Summary

From the above finding, below are the summaries on the findings.

Name / Characteristics

Operating System

Programming Language

Browser

Watir

Windows

Linux

Ruby

Chrome

Firefox

IE

Safari

Mobile Browser

WatiN

Windows

.NET

Firefox

IE

WebDriver

Windows

OS X

Linux

Solaris

.Net

Java

Python

Ruby

Perl

PHP

Chrome

Firefox

IE

Safari

Opera

Mobile Browser

HTTP Object

Windows

.NET

N/A

Figure 7: Automation Selection

Based on the finding above, WebDriver is the most appropriate component for use in this project. Besides that, WebDriver has well online documentation that provides sufficient technical information to assist developer for integration purposes.

Approaches of Designing Web Automation Tool (WAT)

Figure 8: Architecture of Web Automation Tool (WAT)

There are multiple ways an application can design to be executed such as client-server, web or mobile. In order to determine the approaches, it is important to understand the needed of the application such as: -

Is multithreading is required?

Is the application need to be run on web, client-server, or mobile?

How is the environment it need to be executed, such as within or outside of LAN / WAN

Is security concern one of the critical inputs for the product?

When client launch the web automation tool (WAT), it will try to establish connection to the database and at the same time will fetch data from database and display on screen. The application allow software tester to configure the test environment, cases and scenario. During testing, WAT will be automatically launch browser as defined by software tester together with a browser driver which attached to individual browser library from the component.

From the above concern, client-server approaches is the most appropriate because of the nature how the application been executed and threading is required to allow software tester do perform others task simultaneously while executing the test

Client Server Application

In client-server architecture, each computer is connected to a network. Typically, a client is an application that runs on a personal computer or workstation and relies on a server to perform some operations and access remotely via LAN or WAN. For example, an e-mail client is an application that enables you to send and receive e-mail.

Servers are often dedicated, meaning that they perform no other tasks besides their server tasks. On multiprocessing operating systems, however, a single computer can execute several programs at once. A server in this case could refer to the program that is managing resources rather than the entire computer.

Description Purpose of Automation Tool

The purpose of the proposal is to develop a GUI on top of open source component that are capable of provided web automation testing; the application should be able to perform web application testing based on the test case defined and setup in the automation application.

The intended application only modifies the standard HTML object such as textbox or trigger object behaviours. Besides that, it will also capable of provide option to perform simulation and logging mechanism as well.

At the end, it will be able to assist software tester to: -

Configure test case that able to re-execute in future.

Logged result with error code and description in the event of failure.

Able to provide screen capture as prove of testing.

Able to customize ThinkTime by allow software tester to configure a value

Better utilization of resource on testing web application, it can be configure as such to automate the application testing for BackOffice module which is less critical.

Questionnaire

Questionnaire is a research tool that consisting of a series of questions and with the aim of gathering information from targeted audience and it is mostly use for statistical. The conventional approach of questionnaire is to have a face-to-face data gathering. With this technologies era, the faith of questionnaire is changes dramatically from conversional face-to-face approaches to an online medium.

In this project, we there will be two different types of survey been identified and carries out namely: -

Test Automation Survey

Manual vs. Automation Survey

Test Automation Survey

An online questionnaire has been created using a free survey provider "Survey Monkey" (Survey Monkey, 2013). The survey is set with a poll size of 30 IT professional comprising of resources from different roles in a software development cycles in Malaysia.

The purpose of the survey is to understand: -

How well automated testing is accepted in the respective organization

What are the concern when selection an automation tool

How you own it, build or buy?

Expected Return On Investment (ROI)

The survey consist of 8 different question (refer to Appendix B for full detail). Each of the questions has its own purposes. This section we will analyse the result of the survey, and match back with the purpose listed above. Below are the results of survey.

No

Question

Result

1

What is your role in your current organization?

Technical

Functional

Management

56.7%

36.7%

6.7%

2

Do you know what test automation is?

Yes

No

90.0%

10.0%

3

Do you organization use any automation tool currently?

Yes

No

26.7%

73.3%

4

Do you plan to run it in your software development project?

Yes

No

86.7%

13.3%

5

What are the reasons that concern the most when selecting automation tools? (multiple choice)

Support

Feature

Cost

Ease of Use

Customization

53.3%

56.7%

66.7%

50.0%

16.7%

6

What is your preference of owning an automation tool?

Build from scratch

Buy

Not now

36.7%

60.0%

3.3%

7

Test Automation! How much cost saving can it be (1-100)?

81-100

61-80

41-60

21-40

1-20

26.7%

46.7%

20.0%

6.7%

0.0%

8

Do you ever hear of automation tool Selenium, Watir or WatiN?

Yes

No

20.0%

80.0%

Figure 9: Test Automation Survey

Based on the feedback of survey, software organization is Malaysia is well aware of purposes of test automation. However, when look at the implementation of automation, the result is not really encouraging and one of the results might be related to the cost of ownership which co-related to ROI. On the others perspective, software organization is looking forward to implemented it in future project. However I believe with the right education and information on open source, the result of automation adoption could be improved.

Manual vs. Automation Survey

The approach of analysing this section is different from "Test Automation", where it will be carry out by using face-to-face interview with a senior officer of a local bank. The purpose of the survey is to understand the effort required to perform regression testing on the module with minor enhancement.

The following table (test scenario) consisting of actual functionality available in EASY internet banking application. These calculated data representing testing effort needed to perform 1st trial and regression in the subsequent test, and were basically driven by test coverage of each function.

Test Scenario

Manual Test

Automation

Test

1St Attempt

Regression

1st Attempt

Regression

Module: Fund Transfer

Implementation of express fund transfer

5

5

6

0.5

Module: Bill Payment

New biller as merchant

4

5

5

0.5

Module: Notification

Extra notification for payment order

2

2

3

0.5

Module: Reset Password

Change password and history

2

2.5

2

0.5

Module: Account

Additional column to display running amount

1

1.5

4

0.5

Module: Account

Enabled account tagging

2

2.5

3

0.5

Total (Sub Individual)

16

18.5

23

3

Total

34.5

26

Figure 10: Manual vs. Automation (Effort required)

Based on the survey, the effort of perform regression (manual approach) does not speed-up or reduce the testing effort in total. In fact the testing effort will increase over period of times because of new functional be added into the module with purpose of making the application feature rich but at the same time it burden the software tester by allocating more testing effort into the respective modules.

Summary of questionnaire

This section highlighted the 2 important aspect related the test automation, which is: -

Confident level on test automation in software development cycle

Regression using test automation is cost saving.

In test automation survey, it highlighted that 90.0% of the respondent is well aware of what is automation about. 86.7% of respondent is planning to implement automation in their software development but when comes to "concern on selecting automation tool"; the main concern is still cost (66.7%), following by feature (56.7%), support (53.5%) and ease of use (50.0%).

Cost is a primary concern for automation selection, and it may be driven by approach of ownership. If respondent choose to build the automation from scratch, then cost is not an issue whereas if respondent choose to opt by purchasing it, then cost will always the primary concern.

that 86.7% of the respondent plan to implement so automation in the software development.

User Requirement

Analysing the user requirement is an important phase that helps the software developers in specifying system function and is a critical to the success of a software project. The Web Automation Tool should be able to manage following tasks: -

Manage Test Case

Execute Test Scenario

Manage Dynamic Parameter

Reporting

Manage Test Case

WAT should be able to allow software tester to create new test environment and case individually. Created test case should able to support following task: -

Setup dynamic URL for the web application

Capable of retrieving and assigning value to specific HTML element

Capable of simulating user behaviours

Capable of implementing retry mechanism to avoid higher failure rate

Execute Test Scenario

WAT should be able to run or rerun any test scenario defined by software tester. The application also should provide feature to allow simulation to serve as a purposes of verification. This is to avoid error discover during actual test run.

Manage Dynamic Parameter

WAT will be able to allow software tester to configure the value. The configurable value that provided will be used to control customer think time; this is to mimic user response.

Seconds, a retry counter required, it is used to reduce the failure rate of test case. Testing web application is a different concept as compare to client-server application where the response of web application is slower because it goes through the internet cloud, thus it has dependency on the speed of the internet connection.

Reporting

WAT should be has a mechanism to logged the result of testing, furthermore output of testing is all about results. However, the report format is subject to individual thus, at least there are a place where software tester about to receive the records for future processing

Use Case

The Unified Modelling Language (UML) is a language use for specific software system, it helps to describe and visually map a software system design and structure without the boundary of application environment and targeted platform.

A use case diagram look at the system in term of an actor (user), rather than functionality it required to perform. Use Case is the simplest method of representing user interaction of the system and showing different types of users which will be interact with the system.

The actor is the driving force behind the model, an actor is defined as someone who:

"represent a role someone might play, rather than representing a particular individual" (Perdita Stevens, 2003)

Figure 11: Use Case of Automation

The Use Case above has only one actor which is the software tester, and following is the description of the use cases

Tester Login

When the WAT is launch, it is not a prerequisite for software tester to sign-in. However, it is mandatory when software tester choose to execute the test in actual environment. The login information will allowed WAT to create and log the result of executed test cases into database for reporting purposes.

Database Connectivity

By default, the database connection is configured in a configuration file and upon launch; it will load the configuration directly. In the event of failure due, software tester are still able to change the connection during runtime.

Maintain Test Case

Software testers are required to setup information pertaining to test environment include as detail as test data.

Verified Test Case

To making sure that test case is setup correctly, a verification process must be in place. The verification processes can either an eye-ball checking or event performing simulation which available in the automation tools itself.

Execute Test Case

Once everything is setup correctly, software tester can perform an actual or simulation based on the need. However, there are some different between actual and simulation.

Simulation does not save the result of each executed test case and logging because software tester might be using this feature to verify the test case setup. On top of that, the screen capture feature will not be saving the result into picture.

Reset Session

WAT was design only to allow one collection to be executed at any one time. Perfect scenario might not be happen all the times, and naturally application might be crash or hang.

Reset session is design with an intention to act as a backup plan to release the session so that software tester is able to re-execute the same test again.

Reporting

WAT does not design to generate result in any format; however all the result was save into a database. Software tester can pull out the result by either design a pivot to read or export out the result

Maintain Parameter

WAT have few parameters that need to be maintained by software tester, although the package itself will be a default value, however it is advisable for software tester to configure based on the individual environment. Currently there are few parameters namely think time, retry counter and database connection.

Logical Data Model

Logical Data model is the actual implementation and extension of conceptual data model. It represented the business requirement and assumption as the system from a top-down perspective.

From this processes, an Entity Relationship Diagram (ERD) was developed based on the understanding of the system and requirement.

Figure 12: ERD for Automation Tool

The following is the description of the key entities of the ERD for automation. For detail relationship, the evident of entity has been show in section 4.8.

Entity tbl_action

This entity is a system defined action that is not maintainable as of per current. The predefined value consists of: -

Click

Click Link [CSS Selector]

Click Link [LinkText]

Click Link [XPath]

Get Element

Insert

Insert Element

Is Found [Text]

Is Not Found [Text]

Screen Capture

Wait

The fields in the entity contain following information: -

id

action

Entity tbl_environment

This entity will maintain information about the individual web application such as URL. The fields in the entity contain following information: -

id

env_cd

env_dscp

env_url

active

frame

created_dt

Entity tbl_error

This entity use to store the code error that will be will be display on screen. If the code is detected and matched, it will be records in automation log for reporting purposes. The fields in the entity contain following information: -

id

cd

msg

created_dt

Entity tbl_log

This entity use to store the result of actual testing, it will be containing information such as: -

who executed the test

start and end time of specific test scenario

is the test result for specific test case is successful or fail

description of error encounter

The fields in the entity contain following information: -

id

login_id

env_id

session_id

case_id

result

cd

dscp

start_dt

end_dt

Entity tbl_module

This entity stores the logical grouping of test scenario, although the usage of module not really tangible at this moment, however it was design for future expansion. The fields in the entity contain following information: -

id

module_cd

module_dscp

created_dt

Entity tbl_scenario

This entity stores the test scenario which automation will be executing such as "Landing Page - Account Summary". The fields in the entity contain following information: -

id

module_id

scenario_dscp

created_dt

Entity tbl_subscenario

This entity store the steps of scenario defined and it is the most important data that are need by automation. The fields in the entity contain following information: -

id

scenario_id

object_action

object_nm

object_value

thinktime

validateresult

position

created_dt

Entity tbl_test

This entity store the test collection title, it use to display information to software tester about the scenario link to the collection. The fields in the entity contain following information: -

id

env_id

test_dscp

created_dt

Entity tbl_testcase

This entity store the position numbering in which test case will be executed. The fields in the entity contain following information: -

id

test_id

no

scenario_id

created_dt

Entity tbl_user

This entity is also storing identity of software tester, as of now no UI is provided to create new entry. However, it can also maintain using Data Manipulation Language (DML) such as

"INSERT INTO tbl_user (name) VALUES ('Software Tester')"

The fields in the entity contain following information: -

id

name

Relationship of Entity

Entity tbl_action (action)

Entity Action is related to Subscenario by a (1: M) one-to-many relationship.

Entity tbl_environment (environment)

Entity Environment is related to Test by a (1: M) one-to-many relationship, at the same time it also related to Log by a (1: M) one-to-many relationship

Entity tbl_error (error)

Entity Error is related to Log by a (0: M) zero-to-many relationship.

Entity tbl_log (log)

Entity Log is related to Error by a (M: 0) many-to-zero relationship, at the same time it also related to User by a (M: 1) many-to-one relationship and Environment by a (M: 1) many-to-one relationship

Entity tbl_module (module)

Entity Module is related to Scenario by a (1: M) one-to-many relationship.

Entity tbl_scenario (scenario)

Entity Scenario is related to Module by a (M: 1) many-to-one relationship, at the same time it also related to Sub-Scenario by a (1: M) one-to-many relationship and TestCase by a (1: M) one-to-many relationship.

Entity tbl_subscenario (sub-scenario)

Entity Sub-Scenario is related to Action by a (M: 1) many-to-one relationship, at the same time it also related to Log by a (1: M) one-to-many relationship and Scenario by a (M: 1) many-to-one relationship.

Entity tbl_test (test)

Entity Test is related to TestCase by a (1: M) one-to-many relationship, at the same time it also related to Environment by a (M: 1) many-to-one relationship.

Entity tbl_testcase (testcase)

Entity TestCase is related to Scenario by a (1: M) one-to-many relationship, at the same time it also related to Test by a (M: 1) many-to-one relationship.

Entity tbl_user (user)

Entity User is related to log by a (1: M) one-to-many relationship.

Chapter Five: Design

Database Design

Database design is the processes of developing a details data model of a database. This logical data model contains all the needed logical and physical aspect of the structure such as physical storage parameters need to generate a Data Definition Language (DDL).

Based on the WAT design, some table will consist of pre-defined table such as tbl_action, and this records will be created using Data Manipulation Language (DML). DML on the other side will be used for maintaining the records available in the tables.

For this project, the application requires create few tables to act as a based to the application, and the details of the tables can be referring to "Appendix A (Table Structure)"

Design of Graphical User Interface (GUI)

Like many other invention in the history, the idea of GUI were thought long before technology was even available to build it. One of the main researches that express the interest and idea was Vannevar Bush back dated in 1930s (Jeremy Reimer, 2005). Since then, many researches continue the work of improving the GUI and in year 1984 Macintosh breaking the history by providing the first computer with GUI available to publicly (Utah State University, n.d.). GUI allow user to greater accessibility with everything under cursor tip, improve productivity and lower cognitive lode by not required to laid everything out on the spot and remember everything in details.

Today, GUI not only applicable to computing, but it is part of our day-to-day activities. The usage of GUI has been extended to other communication devices such as TV, mobile, refrigerator, microwave and other electronics devices.

Chapter Six: Implementation

Prerequisite to installation

For any software installation, it is important to highlight the requirement for installation to avoid any unforeseen difficulties during execution.

.Net Framework

.Net framework is subjective to be buddle with different version of Windows Operating System. For this project, WAT required .NET 4.0 runtime to be installed in advance before launching the application.

The package can be downloaded for free from Microsoft Download (Microsoft Download, 2013).

Bandwidth Connection

If the environment is accessible via internet cloud, it will be prefer to have a faster internet line with minimum of 1Mbps, this requirement is needed to avoid high rate of failure. However, if the application can be accessible via LAN, then this is not necessary because LAN connections are come with at least 10Mbps.

Internet Browser

WAT is leverage on the installer of internet browser installed in the machine. In this project, WAT will be only supported three main purposes browser which is Internet Explorer, Firefox and Google Chrome (NewRelicBlog, 2012).

Software Tester Skillset

Software testing can be segregate between functional and technical testing. For automation testing, it is prefer to select those with technical knowledge on HTML. The reason it need some technical knowledge is because during the test case creation, software tester need to use some 3rd party developer tool (available to download via add-on) to retrieve the element data.

3rd Party Development Tool

The purposes of acquiring 3rd party development tools is to ease the software tester to easily getting the element value for specific HTML object. The best tool in getting the object name is by using Selenium IDE (Alejandro Brizuela, 2012) which is also one of the software enthusiasms in enhancing the Selenium components.

Web Automation Tool (WAT)

Graphical User Interface for WAT

This section will be projecting the overall mapping of the GUI available in the research of web automation. The application is breaking down into 5 different distinctive access path namely Main Menu, Option, Setup, Test and Help. Below are the skeleton of the web automation.

Figure 13: Overall Menu Architecture

Menu (Main Screen)

Below will be the screen layout when the application launch. It will automatically connect to the database based on database configuration.

When the database is connected, it will be show in [Status] located at bottom left. [Status] will be value of: -

Database is connected

Database is disconnected

Figure 14: Main (Main Screen)

Menu (Option)

Menu (Option) shows the feature that available underneath. The function that available in this menu is "Database Logon" and "User Logon".

Figure 15: Menu (Option)

Option (Database Logon)

Upon activated the screen, it will load the configuration value defined. If the initial connection is fail, software tester is capable of changing the connection during runtimes.

Figure 16: Option (Database Logon)

Option (User Logon)

By default, all the activities within the application does not required user logon. It is required only when executing actual test, because user id is use to log the user for reporting purposes.

Figure 17: Option (User Logon)

Menu (Setup)

Menu (Setup) shows the feature that available underneath. The function that available in this menu is "Environment List", "Test Collection List", "Module List", "Scenario List", "Sub-scenario List" and "Manage Test Collection"

Figure 18: Menu (Setup)

Setup (Environment List)

Environment list allow software tester to create new information of web application environment. Below screen is the listing screen for environment information. The screen listed below allowing software tester to Add, Update and Delete.

Figure 19: Setup (Environment List)

Setup (Test Collection List)

Test collection list defined individual test usage such as the collection will be used for regression, or production testing. This will be used as identified when Test is executed.

Figure 20: Setup (Test Collection List)

Setup (Module List)

Module list is used as a categorisation when perform maintenance on Test Scenario

Figure 21: Setup (Module List)

Setup (Scenario List)

Scenario list is a collection of sub-scenario which defined individual action need to be taken and test.

Figure 22: Setup (Scenario List)

Setup (Sub Scenario List)

Sub-scenario is the most important maintenance which defined what activities to be perform. Currently, there is few action types can choose: -

Action

Description

Click

Simulate user mouse click

Click Link (CSS Selector)

Simulate mouse click on hyperlink via CSS Selector method

Click Link (Link Text)

Simulate mouse click on hyperlink via Link Text

Click Link (XPath)

Simulate mouse click on hyperlink via XPath

Get Element

A feature that allow saving value from element temporary for future processing within the test session

Insert

Simulate user entering data into textbox or text-area

Insert Element

A special dedicated feature to enter OTP

Is Found (Text)

Verified does the specific text is found in the response HTML. If found then function will return TRUE

Is Not Found (Text)

Similar to "Is found (text)", however is return TRUE when text is not found.

Screen Capture

Provide screen capture feature to capture the snapshot of the response HTML

Wait

Simulate user think time

Figure 23: WAT Action List

Figure 24: Setup (Sub Scenario List)

Setup (Manage Test Collection)

Manage test collection is use to maintenance scenario such as insert, delete or change sequence. It is also defined sequence which test needs to execute.

Figure : Setup (Manage Test Collection)

Menu (Test)

Menu (Test) shows the feature that available underneath. The function that available in this menu is "Reset Test Session" and "Execute Test".

Figure : Menu (Test)

Test (Reset Test Scenario)

Application is during in such a way to only allow one Test to be executed at any one-time and will be virtually locking it to prevent other application to be executed.

Due to some unforeseen circumstances and technical difficulties, some browser might be hang or even crash and this is where manual intervention is required to terminate the test session. When it is happen, software tester are required to manually reset using this feature to enabled application to be execute again.

Figure : Menu (Test)

Test (Execute Test)

Application will preload the Test Collection (with sub-scenario) in this screen, and software tester has the option to choose whether to perform an actual test or simulation.

There are some differences between test runs in actual or simulation mode. In actual test, applications allow software testers to defined folder for screen capture, and it enabled logging for statistics purposes. Whereby, simulation only allows application to perform a mock run on the defined test collection.

When a test is executed (regardless of Actual or Simulation),

Figure : Test (Execute Test)

Menu (Help)

Menu (Help) shows the feature that available underneath. The function that available in this menu is "About …"

Figure : Menu (Help)

About

About will provided information about the application and software engineer itself.

Figure 30: About

Configuration

Like any other development, it is not a good practise to hardcoded any value that are deem to be changed under different environment. As mention in section 1.5, the software development will be leverage on .NET platform using Visual Studio 2010 and in such platform parameterize setting by default will be save in app.config field located in the application folder.

There are currently few parameter that required to be configure

Think Time

<setting name="ThinkTime" serializeAs="String">

<value>5000</value>

</setting>

"ThinkTime" is basically is a behaviours of try mimic user action by allowing application to include lapse time for transporting message between client-server by taking into consideration that slow response from the recipient parties. This is important is because if ThinkTime is not set, WAT might just keep requesting web server for the response and it the response is slow or timeout and WAT might threat the cases as fail.

"ThinkTime" value is interrupt by seconds, which 5000 mean 5 seconds.

Retry Counter

<setting name="RetryCounter" serializeAs="String">

<value>3</value>

</setting>

"RetryCount" is to allow WAT to retry when a test case that is fail, WAT will retry up to the value defined in the parameter. If the WAT still not able to execute it successfully, it will be logged into database as fail.

Database Connection

<setting name="DBConnection" serializeAs="String">

<value>Data Source=localhost; Initial Catalog=SeleniumWebDriver; User Id=sa; Password=Passw0rd;</value>

</setting>

"DBConnection" is use to defined the connectivity to automation tool database. By defaulted, application will load the value the try to establish connection. In the event fail, software tester still able to modify the value during runtime, however the value change during runtime will not be save back into configuration file.

Web Service URL (OTP)

<setting name="WebService_OTP" serializeAs="String">

<value>https://mobiletesting.com/misc/OTP.asmx[ [2] ]</value>

</setting>

"WebService_OTP" is a special proprietary integration to a 3rd party application to fetch the OTP code only when software tester creates a test case which will consider high security risk to the EASY Internet Banking system.

Transaction that are consider high risk include all open ended financial transaction, creating customer favourite or changes customer profile information.

Problems Encountered and Overcome

Through the course of design and implementation of this project, there are numerous issues that encountered ranging from requirement, programming to implementation.

Internet Explorer Protected Mode

Internet Explorer by default will only checked "Enabled Protected Mode" for "Internet" and "Restricted sites" only and "Local intranet" and "Trusted sites" remain unchecked.

Figure : IE Security>Option

When the test case design to run on IE, it will not able to launch the processes correctly and it will fail during the initiation processes. In order to solve the issues, make sure to checked "Enabled Protected Mode" for all the four zones and restart all IE session before retest again.

Internet Explorer Trusted Sites

By defaulted when browser navigates to an untrusted URL, it is by design browser will prompt alert and warm about the site.

Figure 32: SSL Certificate Warning

As per current, WebDriver does not have any feature provided to ignore any certificate error for IE.

Figure 33: Added Trusted Site in IE

In order to solve this issue, software tester is required to add the site into Trusted Sites manually.

Internet Explorer 10

Internet Explorer 10 initially offer to public via Windows 8, however Microsoft enhance the IE to be installable into Windows 7 and Windows Server 2008 R2 via service patch. As of now, WebDriver only provide IE compatibility for version 6, 7, 8 and 9. Software tester is advice not to proceed with IE10 testing via automation at this moment until it is stable.

Client Script Interaction

In today technology, there are 2 major different client scripts that are still in use in current technology namely JavaScript and VBScript. JavaScript is native supported by most of the popular internet browser; on the other site VBScript is proprietary only for IE. The implementation of WebDriver on different browser is depending on which engine it is running. As of currently, JavaScript implementation is only supported by Firefox engine and other browser engine.

Due to the scope of work for this project, the development does not cover in handling client scripting and one of the reason is because of standardization of the Action allowed in this project.

Chapter Seven: Testing

In the fast changing world, customer wanted to have their product or services implemented and update faster than their competitors. Customer wanted more for software release with new feature.in shorter time frame without any defect. So due to this short time frame and continuous improvement, bug fixing and verification is very critical.

If software is buggy, it should be release in production environment as it impacts the customer experience which will damage the repetition of the brand or company itself, thus software testing is a must.

Functional Testing Requirement

Functional testing consists of various requirements which are essential to the software tester and system as a whole. The detail of the functionality is as below;

WAT Launch

Case

Expected result

Status

Launch WAP with valid connection string

Status display connected

Pass

Launch WAP with invalid connection string

Display error message

Status display disconnected

Pass

Figure 34: Test Case: WAT launch

Database Connectivity

Case

Expected result

Status

Logon with valid string

Display successful message

Status display connected

Pass

Logon with invalid string

Display error message

Status display disconnected

Pass

Figure 35: Test Case: Database Connectivity

User Logon

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

User Logon

Click Select

Display successful message

Status display connected

Pass

Figure 36: Test Case: User Logon

Maintain Environment List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add environment

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update environment

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete environment

Pass

Delete new created environment

Deleted successfully

Delete existing environment that are in used

Display error message

Figure 37: Test Case: Environment List

Maintain Test Collection List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add test collection

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update test collection

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete test collection

Pass

Delete new test collection

Deleted successfully

Delete existing test collection that are in used

Display error message

Figure 38: Test Case: Test Collection List

Maintain Module List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add module

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update module

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete module

Pass

Delete new module

Deleted successfully

Delete existing module that are in used

Display error message

Figure : Test Case: Module List

Maintain Scenario List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add scenario

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update scenario

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete scenario

Pass

Delete new scenario

Deleted successfully

Delete existing sub-scenario that are in used

Display error message

Figure 40: Test Case: Scenario List

Maintain Sub-Scenario List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add sub-scenario

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Pass

Update sub-scenario

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete sub-scenario

Pass

Delete