Analysis Automation Tool Selection Criteria For Automation

Published Date: 02 Nov 2017

As highlighted in section 1.5 (Technology Overview), Microsoft technology is chosen as the platform for the development of this project as compare to Linux or Mac OS. Below is the reason why such platform is chosen.

Ease of use

Microsoft platform and design is very user-friendly, where it using a single installation package of Windows Installer (MSI) or Executable (EXE) format. The installation process usually required user to click "Next" unless user chose "Advance" installation option.

Integration and Interoperability

As mention in section 1.5, Visual Studio been chosen. The installation step is straight forward together with a set of component available. As compare to it sibling "Eclipse", where setting up the development environment for Mobile and Java development is differently where separate component installation is needed. In separate event, Microsoft has the greatest backward compatibility support.

Documentation

Documentation is an important entity any aspect of the software development cycle, starting from initial requirement to implementation until maintenance. Lacking in documentation might slow down the research and development on the respective area and in order to verified it someone need to perform trial-and-error in making sure that it is really work compatible with existing functional, such trail-and-error is waste of resource if documentation is in place.

Another example, such as implementation documentation is design to assists and smoothen the deployment process, and avoid same mistaken to be happening again.

Analysis of Automation Component Selection

In the open source industry, there are numerous tools that are capable of providing the ability of doing automation. This section will analysis the capabilities of each tool and finalizes the component selection. The following section will provide sufficient information about individual tools

Web Application Testing in Ruby (Watir)

Watir (Watir, n.d.) is an open source (BSD) automation tool running on Ruby that supports Windows and Linux.

Below listed the environments that supported –

Operating System – Windows, Linux

Programing Language - Ruby

Browser – Chrome, Firefox, IE, Safari, Mobile Browser

Documentation – Blog from individual author

Web Application Testing in .NET (WatiN)

WatiN (WatiN, n.d.) is born on December 2005, after inspired by "Watir". It is aim with integrating web application testing possible for .NET languages.

Below listed the environments that supported -

Operating System – Windows

Programing Language - .Net

Browser – Firefox, IE

Documentation – Limited resource

SeleniumHQ

Originally, start in 2004 in ThoughtWork (SeleniumHQ, n.d.) in Chicago as "JavaScriptTestRunner" for testing internal application. The workgroup later expanded outside of ThoughtWork with researches located remotely across the worlds such as Japan and China.

WebDriver is a codename for one of its many components and is capable of supporting large number of browser vendor who have taken steps to make WebDriver a natives part of their browser. It is also the core technology in countless other browser automation tool, API and frameworks. This is because of WebDriver have a large number of contributor and good documentation (WebDriver, 2013).

Below listed the environment that supported -

Operating System – Windows, OS X, Linux, Solaris

Programing Language - .Net, Java, Python, Ruby, Perl, PHP

Browser – Chrome, Firefox, IE, Safari, Opera, Mobile Browser

Documentation – Blog from author and object reference (WebDriver, 2013)

HTTP Object

HTTP Request and Response (Connolly, 2004) is natively support by most languages such as Java, .NET, Ruby and others. The object is use for transporting request via tunnel of HTTP from client to server or via versa. Because it is leverage on the transport level, thus it is making the implementation harder and required skillset on transport level development.

Below listed the environments that supported -

Operating System – Windows

Programing Language - .Net, Java, Python, Ruby, Perl, PHP

Browser – N/A

Documentation – limited resource

Summary

From the above finding, below are the summaries on the findings.

Name / Characteristics

Operating System

Programming Language

Browser

Watir

Windows

Linux

Ruby

Chrome

Firefox

IE

Safari

Mobile Browser

WatiN

Windows

.NET

Firefox

IE

WebDriver

Windows

OS X

Linux

Solaris

.Net

Java

Python

Ruby

Perl

PHP

Chrome

Firefox

IE

Safari

Opera

Mobile Browser

HTTP Object

Windows

.NET

N/A

Figure 1: Automation Selection

Based on the finding above, WebDriver is the most appropriate component for use in this project.

As mention above, selenium have advantage over well organize online documentation that provides sufficient technical information to assist developer for integration purposes, and publicly available via the website (Selenium, 2013).

Approaches of Designing Web Automation Tool (WAT)

Figure 2: Architecture of Web Automation Tool (WAT)

There are multiple ways an application can design to be executed such as client-server, web or mobile. In order to determine the approaches, it is important to understand the needed of the application such as: -

Is multithreading is required?

Is the application need to be run on web, client-server, or mobile?

How is the environment it need to be executed, such as within or outside of LAN / WAN

Is security concern one of the critical inputs for the product?

When client launch the web automation tool (WAT), it will try to establish connection to the database and at the same time will fetch data from database and display on screen. The application allow software tester to configure the test environment, cases and scenario. During testing, WAT will be automatically launch browser as defined by software tester together with a browser driver which attached to individual browser library from the component.

From the above concern, client-server approaches is the most appropriate because of the nature how the application been executed and threading is required to allow software tester do perform others task simultaneously while executing the test

Client Server Application

In client-server architecture, each computer is connected to a network. Typically, a client is an application that runs on a personal computer or workstation and relies on a server to perform some operations and access remotely via LAN or WAN. For example, an e-mail client is an application that enables you to send and receive e-mail.

The servers are normally dedicated machine use to serve the task assign from the caller application. In a multi-processor operating system environment, a server is able to consume multiple tasks at any single instant of time.

Description Purpose of Automation Tool

The purpose of the proposal is to develop a GUI on top of open source component that are capable of provided web automation testing; the application should be able to perform web application testing based on the test case defined and setup in the automation application.

The intended application only modifies the standard HTML object such as textbox or trigger object behaviours. Besides that, it will also capable of provide option to perform simulation and logging mechanism as well.

At the end, it will be able to assist software tester to: -

Configure test case that able to re-execute in future.

Logged result with error code and description in the event of failure.

Able to provide screen capture as prove of testing.

Able to customize ThinkTime by allow software tester to configure a value

Better utilization of resource on testing web application, it can be configure as such to automate the application testing for BackOffice module which is less critical.

Questionnaire

Questionnaire is a research tool that consisting of a series of questions and with the aim of gathering information from targeted audience and it is mostly use for statistical. The conventional approach of questionnaire is to have a face-to-face data gathering. With this technologies era, the faith of questionnaire is changes dramatically from conversional face-to-face approaches to an online medium.

Test Automation Survey

An online questionnaire has been created using a free survey provider "Survey Monkey" (Survey Monkey, 2013). The survey is set with a poll size of 30 IT professional comprising of resources from different roles in a software development cycles in Malaysia.

The purpose of the survey is to understand: -

How well automated testing is accepted in the respective organization

What are the concern when selection an automation tool

How you own it, build or buy?

Expected Return On Investment (ROI)

The survey consist of 8 different question (refer to Appendix B for full detail). Each of the questions has its own purposes. This section we will analyse the result of the survey, and match back with the purpose listed above. Below are the results of survey.

No

Question

Result

1

What is your role in your current organization?

Technical

Functional

Management

56.7%

36.7%

6.7%

2

Do you know what test automation is?

Yes

No

90.0%

10.0%

3

Do you organization use any automation tool currently?

Yes

No

26.7%

73.3%

4

Do you plan to run it in your software development project?

Yes

No

86.7%

13.3%

5

What are the reasons that concern the most when selecting automation tools? (multiple choice)

Support

Feature

Cost

Ease of Use

Customization

53.3%

56.7%

66.7%

50.0%

16.7%

6

What is your preference of owning an automation tool?

Build from scratch

Buy

Not now

36.7%

60.0%

3.3%

7

Test Automation! How much cost saving can it be (1-100)?

81-100

61-80

41-60

21-40

1-20

26.7%

46.7%

20.0%

6.7%

0.0%

8

Do you ever hear of automation tool Selenium, Watir or WatiN?

Yes

No

20.0%

80.0%

Figure 3: Test Automation Survey

Based on the feedback of survey, software organization is Malaysia is well aware of purposes of test automation. However, when look at the implementation of automation, the result is not really encouraging and one of the results might be related to the cost of ownership which co-related to ROI. On the others perspective, software organization is looking forward to implemented it in future project. However I believe with the right education and information on open source, the result of automation adoption could be improved.

Summary of questionnaire

This section highlighted the 2 important aspect related the test automation, which is: -

Confident level on test automation in software development cycle

Regression using test automation is cost saving.

In test automation survey, it highlighted that 90.0% of the respondent is well aware of what is automation about. 86.7% of respondent is planning to implement automation in their software development but when comes to "concern on selecting automation tool"; the main concern is still cost (66.7%), following by feature (56.7%), support (53.5%) and ease of use (50.0%). Cost is a primary concern for automation selection, and it may be driven by approach of ownership. If respondent choose to build the automation from scratch, then cost is not an issue whereas if respondent choose to opt by purchasing it, then cost will always the primary concern.

User Requirement

Analysing the user requirement is an important phase that helps the software developers in specifying system function and is a critical to the success of a software project. The Web Automation Tool should be able to manage following tasks: -

Manage Test Case

Execute Test Scenario

Manage Dynamic Parameter

Reporting

Manage Test Case

WAT should be able to allow software tester to create new test environment and case individually. Created test case should able to support following task: -

Setup dynamic URL for the web application

Capable of retrieving and assigning value to specific HTML element

Capable of simulating user behaviours

Capable of implementing retry mechanism to avoid higher failure rate

Execute Test Scenario

WAT should be able to run or rerun any test scenario defined by software tester. The application also should provide feature to allow simulation to serve as a purposes of verification. This is to avoid error discover during actual test run.

Manage Dynamic Parameter

WAT will be able to allow software tester to configure the value. The configurable value that provided will be used to control customer think time; this is to mimic user response.

Seconds, a retry counter required, it is used to reduce the failure rate of test case. Testing web application is a different concept as compare to client-server application where the response of web application is slower because it goes through the internet cloud, thus it has dependency on the speed of the internet connection.

Reporting

WAT should be has a mechanism to logged the result of testing, furthermore output of testing is all about results. However, the report format is subject to individual thus, at least there are a place where software tester about to receive the records for future processing

Modelling

In this modelling section, we will be using Unified Modelling Language (UML) to further assist analysis on the requirement. UML is a combine’s technique from data, business and object modelling and it has been implemented across different technologies throughout the software development life cycle.

Class Diagram

Class diagram is part of the UML that describe static structure diagram that describes the structure of a system by showing the classes, attribute, operation and relationship among them. In this Web Automation, there are 2 solution namely "SeleniumWebDriver" and "ebSeleniumCommon".

"SeleniumWebDriver" does not contain any classes and mainly consist of User Interface which use to maintain user data creation, whereas "ebSeleniumCommon" is the portion that consist to classes use to interact with interface and other important activities such as performing testing driver by test scenario.

Figure 4: Class Diagram for ebSeleniumCommon

Figure 11, show illustrates those identified as important and listed as below:

ebVariable

ebStoredProc

ebWebDriver

ebCommon

ebMessage

The Core working classes of the system are ebCommon and ebWebDriver. ebCommon will consist of function mainly related to database such as establish database connection and besides that, it also have a function fetch OTP.

ebWebDriver on the other hand, is the responsible for creating the WebDriver object for internet browser initialization and most importantly is to execute the test scenario based on sequence set by software testers.

Sequence Diagram

Sequence diagram are used mainly for documenting design, validate the architecture of the system by demonstrate the sequence of actions required to perform a single task or scenario. It is a useful design tools that provide information on dynamic view of the system behaviour which can be difficult to pin point on static specification.

Sequence diagram are typically used to explain object-oriented software systems, whereby it sequence identified the flow of scenario and request underlying it order. Besides that, it also clearly stated object and taken need to be taken by each entities.

Figure 5: Sequence Diagram for Maintain and Execute Test Case

The above section will highlight the basic activities to perform an end-to-end transaction from initialization of the application, to execution of test case.

Firstly, software tester trigger the Web Automation Tool and it will automatically connect to database based on the setting. If the database connection is not established, an error message "Connection to Selenium database is not available. Please connect it manually" will be prompt.

Secondly, software tester will proceed with setting up mandatory information such as Environment, Test Collection, Module, Scenario and Test Scenario.

Thirdly, once the software tester complete the configuration it is good to go. Software tester just needs to choose which test scenario that needs to be executed with option to run as actual or simulation.

Finally, software tester is able to extract log if the test were executed as actual.

Use Case Diagram

The Unified Modelling Language (UML) is a language use for specific software system, it helps to describe and visually map a software system design and structure without the boundary of application environment and targeted platform.

A use case diagram look at the system in term of an actor (user), rather than functionality it required to perform. Use Case is the simplest method of representing user interaction of the system and showing different types of users which will be interact with the system.

The actor is the driving force behind the model, an actor is defined as someone who:

"Represent a role someone might play, rather than representing a particular individual" (Perdita Stevens, 2003)

Figure 6: Use Case of Automation

The Use Case above has only one actor which is the software tester, and following is the description of the use cases

Tester Login

When the WAT is launch, it is not a prerequisite for software tester to sign-in. However, it is mandatory when software tester choose to execute the test in actual environment. The login information will allowed WAT to create and log the result of executed test cases into database for reporting purposes.

Database Connectivity

By default, the database connection is configured in a configuration file and upon launch; it will load the configuration directly. In the event of failure due, software tester are still able to change the connection during runtime.

Maintain Test Case

Software testers are required to setup information pertaining to test environment include as detail as test data.

Verified Test Case

To making sure that test case is setup correctly, a verification process must be in place. The verification processes can either an eye-ball checking or event performing simulation which available in the automation tools itself.

Execute Test Case

Once everything is setup correctly, software tester can perform an actual or simulation based on the need. However, there are some different between actual and simulation.

Simulation does not save the result of each executed test case and logging because software tester might be using this feature to verify the test case setup. On top of that, the screen capture feature will not be saving the result into picture.

Reset Session

WAT was design only to allow one collection to be executed at any one time. Perfect scenario might not be happen all the times, and naturally application might be crash or hang.

Reset session is design with an intention to act as a backup plan to release the session so that software tester is able to re-execute the same test again.

Reporting

WAT does not design to generate result in any format; however all the result was save into a database. Software tester can pull out the result by either design a pivot to read or export out the result

Maintain Parameter

WAT have few parameters that need to be maintained by software tester, although the package itself will be a default value, however it is advisable for software tester to configure based on the individual environment. Currently there are few parameters namely think time, retry counter and database connection.

Logical Data Model

Logical Data model is the actual implementation and extension of conceptual data model. It represented the business requirement and assumption as the system from a top-down perspective.

From this processes, an Entity Relationship Diagram (ERD) was developed based on the understanding of the system and requirement.

Figure 7: ERD for Automation Tool

The following is the description of the key entities of the ERD for automation. For detail relationship, the evident of entity has been show in section 4.8.

Entity tbl_action

This entity is a system defined action that is not maintainable as of per current. The predefined value consists of: -

Click

Click Link [CSS Selector]

Click Link [LinkText]

Click Link [XPath]

Get Element

Insert

Insert Element

Is Found [Text]

Is Not Found [Text]

Screen Capture

Wait

The fields in the entity contain following information: -

id

action

Entity tbl_environment

This entity will maintain information about the individual web application such as URL. The fields in the entity contain following information: -

id

env_cd

env_dscp

env_url

active

frame

created_dt

Entity tbl_error

This entity use to store the code error that will be will be display on screen. If the code is detected and matched, it will be records in automation log for reporting purposes. The fields in the entity contain following information: -

id

cd

msg

created_dt

Entity tbl_log

This entity use to store the result of actual testing, it will be containing information such as: -

who executed the test

start and end time of specific test scenario

is the test result for specific test case is successful or fail

description of error encounter

The fields in the entity contain following information: -

id

login_id

env_id

session_id

case_id

result

cd

dscp

start_dt

end_dt

Entity tbl_module

This entity stores the logical grouping of test scenario, although the usage of module not really tangible at this moment, however it was design for future expansion. The fields in the entity contain following information: -

id

module_cd

module_dscp

created_dt

Entity tbl_scenario

This entity stores the test scenario which automation will be executing such as "Landing Page - Account Summary". The fields in the entity contain following information: -

id

module_id

scenario_dscp

created_dt

Entity tbl_subscenario

This entity store the steps of scenario defined and it is the most important data that are need by automation. The fields in the entity contain following information: -

id

scenario_id

object_action

object_nm

object_value

thinktime

validateresult

position

created_dt

Entity tbl_test

This entity store the test collection title, it use to display information to software tester about the scenario link to the collection. The fields in the entity contain following information: -

id

env_id

test_dscp

created_dt

Entity tbl_testcase

This entity store the position numbering in which test case will be executed. The fields in the entity contain following information: -

id

test_id

no

scenario_id

created_dt

Entity tbl_user

This entity is also storing identity of software tester, as of now no UI is provided to create new entry. However, it can also maintain using Data Manipulation Language (DML) such as

"INSERT INTO tbl_user (name) VALUES ('Software Tester')"

The fields in the entity contain following information: -

id

name

Relationship of Entity Relationship Diagram

Entity tbl_action (action)

Entity Action is related to Subscenario by a (1: M) one-to-many relationship.

Entity tbl_environment (environment)

Entity Environment is related to Test by a (1: M) one-to-many relationship, at the same time it also related to Log by a (1: M) one-to-many relationship

Entity tbl_error (error)

Entity Error is related to Log by a (0: M) zero-to-many relationship.

Entity tbl_log (log)

Entity Log is related to Error by a (M: 0) many-to-zero relationship, at the same time it also related to User by a (M: 1) many-to-one relationship and Environment by a (M: 1) many-to-one relationship

Entity tbl_module (module)

Entity Module is related to Scenario by a (1: M) one-to-many relationship.

Entity tbl_scenario (scenario)

Entity Scenario is related to Module by a (M: 1) many-to-one relationship, at the same time it also related to Sub-Scenario by a (1: M) one-to-many relationship and TestCase by a (1: M) one-to-many relationship.

Entity tbl_subscenario (sub-scenario)

Entity Sub-Scenario is related to Action by a (M: 1) many-to-one relationship, at the same time it also related to Log by a (1: M) one-to-many relationship and Scenario by a (M: 1) many-to-one relationship.

Entity tbl_test (test)

Entity Test is related to TestCase by a (1: M) one-to-many relationship, at the same time it also related to Environment by a (M: 1) many-to-one relationship.

Entity tbl_testcase (testcase)

Entity TestCase is related to Scenario by a (1: M) one-to-many relationship, at the same time it also related to Test by a (M: 1) many-to-one relationship.

Entity tbl_user (user)

Entity User is related to log by a (1: M) one-to-many relationship.

Chapter Five: Design

Database Design

Database design is the processes of developing a details data model of a database. This logical data model contains all the needed logical and physical aspect of the structure such as physical storage parameters need to generate a Data Definition Language (DDL).

Based on the WAT design, some table will consist of pre-defined table such as tbl_action, and this records will be created using Data Manipulation Language (DML). DML on the other side will be used for maintaining the records available in the tables.

For this project, the application requires create few tables to act as a based to the application, and the details of the tables can be referring to "Appendix A (Table Structure)"

Design of Graphical User Interface (GUI)

Like many other invention in the history, the idea of GUI were thought long before technology was even available to build it. One of the main researches that express the interest and idea was Vannevar Bush back dated in 1930s (Jeremy Reimer, 2005). Since then, many researches continue the work of improving the GUI and in year 1984 Macintosh breaking the history by providing the first computer with GUI available to publicly (Utah State University, n.d.). GUI allow user to greater accessibility with everything under cursor tip, improve productivity and lower cognitive lode by not required to laid everything out on the spot and remember everything in details.

Today, GUI not only applicable to computing, but it is part of our day-to-day activities. The usage of GUI has been extended to other communication devices such as TV, mobile, refrigerator, microwave and other electronics devices.

Chapter Six: Implementation

Prerequisite to installation

For any software installation, it is important to highlight the requirement for installation to avoid any unforeseen difficulties during execution. For installation guide, please refer to Appendix C.

.Net Framework

.Net framework is subjective to be buddle with different version of Windows Operating System. For this project, WAT required .NET 4.0 runtime to be installed in advance before launching the application. The package can be downloaded for free from Microsoft Download (Microsoft Download, 2013).

Bandwidth Connection

If the environment is accessible via internet cloud, it will be prefer to have a faster internet line with minimum of 1Mbps, this requirement is needed to avoid high rate of failure. However, if the application can be accessible via LAN, then this is not necessary because LAN connections are come with at least 10Mbps.

Internet Browser

WAT is leverage on the installer of internet browser installed in the machine. In this project, WAT will be only supported three main purposes browser which is Internet Explorer, Firefox and Google Chrome (NewRelicBlog, 2012). To perform testing using WAT, please installed internet browser stated below:

Firefox – version 20.0.1

Internet Explorer – version 8.0.7601.17514

Chrome – version 26.0.1410.64

Software Tester Skillset

Software testing can be segregate between functional and technical testing. For automation testing, it is prefer to select those with technical knowledge on HTML. The reason it need some technical knowledge is because during the test case creation, software tester need to use some 3rd party developer tool (available to download via add-on) to retrieve the element data.

3rd Party Development Tool

The purposes of acquiring 3rd party development tools is to ease the software tester to easily getting the element value for specific HTML object. The best tool in getting the object name is by using Selenium IDE (Alejandro Brizuela, 2012) which is also one of the software enthusiasms in enhancing the Selenium components.

Web Automation Tool (WAT)

Graphical User Interface for WAT

This section will be projecting the overall mapping of the GUI available in the research of web automation. The application is breaking down into 5 different distinctive access path namely Main Menu, Option, Setup, Test and Help. Below are the skeleton of the web automation.

Figure 8: Overall Menu Architecture

Menu (Main Screen)

Below will be the screen layout when the application launch. It will automatically connect to the database based on database configuration.

When the database is connected, it will be show in [Status] located at bottom left. [Status] will be value of: -

Database is connected

Database is disconnected

Figure 9: Main (Main Screen)

Menu (Option)

Menu (Option) shows the feature that available underneath. The function that available in this menu is "Database Logon" and "User Logon".

Figure 10: Menu (Option)

Option (Database Logon)

Upon activated the screen, it will load the configuration value defined. If the initial connection is fail, software tester is capable of changing the connection during runtimes.

Figure 11: Option (Database Logon)

Option (User Logon)

By default, all the activities within the application does not required user logon. It is required only when executing actual test, because user id is use to log the user for reporting purposes.

Figure 12: Option (User Logon)

Menu (Setup)

Menu (Setup) shows the feature that available underneath. The function that available in this menu is "Environment List", "Test Collection List", "Module List", "Scenario List", "Sub-scenario List" and "Manage Test Collection"

Figure 13: Menu (Setup)

Setup (Environment List)

Environment list allow software tester to create new information of web application environment. Below screen is the listing screen for environment information. The screen listed below allowing software tester to Add, Update and Delete.

Figure 14: Setup (Environment List)

Setup (Test Collection List)

Test collection list defined individual test usage such as the collection will be used for regression, or production testing. This will be used as identified when Test is executed.

Figure 15: Setup (Test Collection List)

Setup (Module List)

Module list is used as a categorisation when perform maintenance on Test Scenario

Figure 16: Setup (Module List)

Setup (Scenario List)

Scenario list is a collection of sub-scenario which defined individual action need to be taken and test.

Figure 17: Setup (Scenario List)

Setup (Sub Scenario List)

Sub-scenario is the most important maintenance which defined what activities to be perform. Currently, there is few action types can choose: -

Action

Description

Click

Simulate user mouse click

Click Link (CSS Selector)

Simulate mouse click on hyperlink via CSS Selector method

Click Link (Link Text)

Simulate mouse click on hyperlink via Link Text

Click Link (XPath)

Simulate mouse click on hyperlink via XPath

Get Element

A feature that allow saving value from element temporary for future processing within the test session

Insert

Simulate user entering data into textbox or text-area

Insert Element

A special dedicated feature to enter OTP

Is Found (Text)

Verified does the specific text is found in the response HTML. If found then function will return TRUE

Is Not Found (Text)

Similar to "Is found (text)", however is return TRUE when text is not found.

Screen Capture

Provide screen capture feature to capture the snapshot of the response HTML

Wait

Simulate user think time

Figure 18: WAT Action List

Figure 19: Setup (Sub Scenario List)

Setup (Manage Test Collection)

Manage test collection is use to maintenance scenario such as insert, delete or change sequence. It is also defined sequence which test needs to execute.

Figure 20: Setup (Manage Test Collection)

Menu (Test)

Menu (Test) shows the feature that available underneath. The function that available in this menu is "Reset Test Session" and "Execute Test".

Figure 21: Menu (Test)

Test (Reset Test Scenario)

Application is during in such a way to only allow one Test to be executed at any one-time and will be virtually locking it to prevent other application to be executed.

Due to some unforeseen circumstances and technical difficulties, some browser might be hang or even crash and this is where manual intervention is required to terminate the test session. When it is happen, software tester are required to manually reset using this feature to enabled application to be execute again.

Figure 22: Menu (Test)

Test (Execute Test)

Application will preload the Test Collection (with sub-scenario) in this screen, and software tester has the option to choose whether to perform an actual test or simulation.

There are some differences between test runs in actual or simulation mode. In actual test, applications allow software testers to defined folder for screen capture, and it enabled logging for statistics purposes. Whereby, simulation only allows application to perform a mock run on the defined test collection.

When a test is executed (regardless of Actual or Simulation),

Figure 23: Test (Execute Test)

Menu (Help)

Menu (Help) shows the feature that available underneath. The function that available in this menu is "About …"

Figure 24: Menu (Help)

About

About will provided information about the application and software engineer itself.

Figure 25: About

Configuration

Like any other development, it is not a good practise to hardcoded any value that are deem to be changed under different environment. As mention in section 1.5, the software development will be leverage on .NET platform using Visual Studio 2010 and in such platform parameterize setting by default will be save in app.config field located in the application folder. There are currently few parameter that required to be configure

Think Time

<setting name="ThinkTime" serializeAs="String">

<value>5000</value>

</setting>

"ThinkTime" is basically is a behaviours of try mimic user action by allowing application to include lapse time for transporting message between client-server by taking into consideration that slow response from the recipient parties. This is important is because if ThinkTime is not set, WAT might just keep requesting web server for the response and it the response is slow or timeout and WAT might threat the cases as fail. "ThinkTime" value is interrupt by seconds, which 5000 mean 5 seconds.

Function SetThinkTime(ByVal objVariable As ebSeleniumCommon.ebVariable, _

Optional ByVal ThinkTime As Integer = 0) As Boolean

SetThinkTime = False

Try

If ThinkTime > 0 Then

objWebDriver.Manage.Timeouts.ImplicitlyWait_

(TimeSpan.FromMilliseconds(ThinkTime))

Else

Thread.Sleep(My.Settings.ThinkTime)

End If

SetThinkTime = True

Catch ex As System.Exception

End Try

End Function

Above function is the routine that mimic user behaviours

Retry Counter

<setting name="RetryCounter" serializeAs="String">

<value>3</value>

</setting>

"RetryCount" is to allow WAT to retry when a test case that is fail, WAT will retry up to the value defined in the parameter. If the WAT still not able to execute it successfully, it will be logged into database as fail.

Database Connection

<setting name="DBConnection" serializeAs="String">

<value>Data Source=localhost; Initial Catalog=SeleniumWebDriver; User Id=sa; Password=Passw0rd;</value>

</setting>

"DBConnection" is use to defined the connectivity to automation tool database. By defaulted, application will load the value the try to establish connection. In the event fail, software tester still able to modify the value during runtime, however the value change during runtime will not be save back into configuration file.

Function SetDatabaseConnection(ByRef objConn As SqlClient.SqlConnection, _

ByVal ConnectionString As String, _

Optional ByVal ErrMessage As String = "", _

Optional ByVal ApplicationTitle As String = CONST_APP_NAME) _

As SqlClient.SqlConnection

SetDatabaseConnection = objConn

Try

objConn = New SqlClient.SqlConnection(ConnectionString)

objConn.Open()

If objConn.State = ConnectionState.Open Then

SetDatabaseConnection = objConn

End If

Catch ex As Exception

If (ErrMessage.Length > 0) And (ApplicationTitle.Length > 0) Then

MsgBox(ErrMessage, _

Microsoft.VisualBasic.MsgBoxStyle.Critical, ApplicationTitle)

Else

MsgBox(ex.Message, MsgBoxStyle.Critical, CONST_APP_NAME)

End If

End Try

End Function

Above function is the routine that handling database connection for the entire WAT.

Web Service URL (OTP)

<setting name="WebService_OTP" serializeAs="String">

<value>https://mobiletesting.com/misc/OTP.asmx[ [1] ]</value>

</setting>

"WebService_OTP" is a special proprietary integration to a 3rd party application to fetch the OTP code only applicable to test case which will consider high security risk to the EASY Internet Banking system. Transaction that are consider high risk include all open ended financial transaction, creating customer favourite or changes customer profile information.

Function GetOTP(ByVal SecuriyCode As String) As String

Dim objOTP As New WebService_OTP.Service

'GetOTPDate - a function that return today date in OTP format

Try

GetOTP = objOTP.GetOTP(SecuriyCode, GetOTPDate)

Catch ex As Exception

GetOTP = String.Empty

End Try

End Function

Above function try to establish a Web Service called "GetOTP" to with "SecurityCode" and "GetOTPDate" as a parameter, if exception does happen it will return "String.Empty" back.

Problems Encountered and Overcome

Through the course of design and implementation of this project, there are numerous issues that encountered ranging from requirement, programming to implementation.

Internet Explorer Protected Mode

Internet Explorer by default will only checked "Enabled Protected Mode" for "Internet" and "Restricted sites" only and "Local intranet" and "Trusted sites" remain unchecked.

Figure 26: IE Security>Option

When the test case design to run on IE, it will not able to launch the processes correctly and it will fail during the initiation processes. In order to solve the issues, make sure to checked "Enabled Protected Mode" for all the four zones and restart all IE session before retest again.

Internet Explorer Trusted Sites

By defaulted when browser navigates to an untrusted URL, it is by design browser will prompt alert and warm about the site.

Figure 27: SSL Certificate Warning

As per current, WebDriver does not have any feature provided to ignore any certificate error for IE.

Figure 28: Added Trusted Site in IE

In order to solve this issue, software tester is required to add the site into Trusted Sites manually.

Internet Explorer 10

Internet Explorer 10 initially offer to public via Windows 8, however Microsoft enhance the IE to be installable into Windows 7 and Windows Server 2008 R2 via service patch. As of now, WebDriver only provide IE compatibility for version 6, 7, 8 and 9. Software tester is advice not to proceed with IE10 testing via automation at this moment until it is stable.

Client Script Interaction

In today technology, there are 2 major different client scripts that are still in use in current technology namely JavaScript and VBScript. JavaScript is native supported by most of the popular internet browser; on the other site VBScript is proprietary only for IE. The implementation of WebDriver on different browser is depending on which engine it is running. As of currently, JavaScript implementation is only supported by Firefox engine and other browser engine.

Due to the scope of work for this project, the development does not cover in handling client scripting and one of the reason is because of standardization of the Action allowed in this project.

Chapter Seven: Testing

In the fast changing world, customer wanted to have their product or services implemented and update faster than their competitors. Customer wanted more for software release with new feature.in shorter time frame without any defect. So due to this short time frame and continuous improvement, bug fixing and verification is very critical.

If software is buggy, it should be release in production environment as it impacts the customer experience which will damage the repetition of the brand or company itself, thus software testing is a must.

Functional Testing Requirement

Functional testing consists of various requirements which are essential to the software tester and system as a whole. The detail of the functionality is as below;

WAT Launch

Case

Expected result

Status

Launch WAP with valid connection string

Status display connected

Pass

Launch WAP with invalid connection string

Display error message

Status display disconnected

Pass

Figure 29: Test Case: WAT launch

Database Connectivity

Case

Expected result

Status

Logon with valid string

Display successful message

Status display connected

Pass

Logon with invalid string

Display error message

Status display disconnected

Pass

Figure 30: Test Case: Database Connectivity

User Logon

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

User Logon

Click Select

Display successful message

Status display connected

Pass

Figure 31: Test Case: User Logon

Maintain Environment List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add environment

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update environment

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete environment

Pass

Delete new created environment

Deleted successfully

Delete existing environment that are in used

Display error message

Figure 32: Test Case: Environment List

Maintain Test Collection List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add test collection

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update test collection

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete test collection

Pass

Delete new test collection

Deleted successfully

Delete existing test collection that are in used

Display error message

Figure 33: Test Case: Test Collection List

Maintain Module List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add module

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update module

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete module

Pass

Delete new module

Deleted successfully

Delete existing module that are in used

Display error message

Figure 34: Test Case: Module List

Maintain Scenario List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add scenario

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update scenario

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete scenario

Pass

Delete new scenario

Deleted successfully

Delete existing sub-scenario that are in used

Display error message

Figure 35: Test Case: Scenario List

Maintain Sub-Scenario List

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add sub-scenario

Pass

Entry valid detail in all fields

Display successful message

Entry partial detail in fields

Display error message

Update sub-scenario

Pass

Modify valid detail in all fields

Display successful message

Modify partial detail in fields

Display error message

Delete sub-scenario

Pass

Delete new sub-scenario

Deleted successfully

Delete existing sub-scenario that are in used

Display error message

Figure 36: Test Case: Sub-Scenario List

Manage Test Collection

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Add scenario

Display successful message

Pass

Delete sub-scenario

Display successful message

Pass

Change test case sequence

Display successful message

Pass

Figure 37: Test Case: Test Collection

Reset Test Session

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Reset Session

Display successful message

Pass

Figure 38: Test Case: Reset Test Session

Execute Test

Case

Expected result

Status

No database connection

Display error message

Status display disconnected

Pass

Start Testing

Pass

Prompt for user login

Alert display correctly

Prompt for output folder

Alert display correctly

Screen Capture

Screen capture store in output directory

Logging

Log is created in tbl_log

Test session is lock

Only allow one test session at one time

Simulation

Start without any alert

Pass

Figure 39: Test Case: Execute Test

Non-Functional Testing Requirement

Non-functional testing is a type of testing of software application for its non-functional requirement. The names of many non-functional tests are often used interchangeably because of the overlap in scope between various non-functional requirements.

Compatibility Test

In today’s world, end users can choose from a number of browsers, operating systems and technology stacks and it is expected that the applications they use are compatible with these. Compatibility Testing helps in verifying that the applications work as expected across different browsers, operating systems and technology stacks. Browser compatibility is one of the many features available in WAT.

Robustness Test

"the degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions" (IEEE, 2010)

The goal of robustness testing is to develop test case environments where a robustness can be assessed. The process is basically in making sure that the progream is tesable, if error is discover in the middle of testing, it need to be log or prompt and avoid application been hang or crash.

Ease of Used

The system is design is such a way that it is navigation user-friendly and lack of complexity. It particularly proven that all transaction performed effortless because all the privilege embedded in the logic thereby allowing users to carry only action they should be capable.

Testing Comparison: Manual vs. Automation

The purpose of this section is to understand how well the testing can peform as compare between manual and automated.

The following table (test scenario) consisting of actual functionality available in EASY internet banking application. These calculated data representing testing effort needed to perform 1st trial and regression in the subsequent test, and were basically driven by test coverage of each function.

Test Scenario

Manual Test

Automation

Test

1St Attempt

Regression

1st Attempt

Regression

Module: Fund Transfer

Implementation of express fund transfer

5

5

6

0.5

Module: Bill Payment

New biller as merchant

4

5

5

0.5

Module: Notification

Extra notification for payment order

2

2

3

0.5

Module: Reset Password

Change password and history

2

2.5

2

0.5

Module: Account

Additional column to display running amount

1

1.5

4

0.5

Module: Account

Enabled account tagging

2

2.5

3

0.5

Total (Sub Individual) (in days)

16

18.5

23

3

Total (in days)

34.5

26

Figure 40: Manual vs. Automation (Effort required)

Based on the survey, the effort of perform regression (manual approach) does not speed-up or reduce the testing effort in total. In fact the testing effort will increase over period of times because of new functional be added into the module with purpose of making the application feature rich but at the same time it burden the software tester by allocating more testing effort into the respective modules.

In this comparison, it highlighted that the effort required performing 1st attempt and regression testing using both manual and automation. In comparison to both, testing resource for manual testing (1st attempt) is shorter than automation but however during regression testing, automation show a large significant different as compare to manual testing. When compare between manual and automated as a whole, automated does have a saving of 8.5 days.

Chapter Eight: Conclusion and Recommendation

This chapter provide major conclusion and recommendation of the report. This investigation has convince and proven that Web Automation Tool can be delivered through a client-server environment. The technology adopted for this project work were found to be suitable to a reasonable level compared to other technologies which were reviewed, the project however has archived it objectives, which end result is to produce a Web Automation Tools. Together with the submission, it will accompany by a demonstration of the final product "Web Automation Tool".

Project achievement

A quick refresh back on the objective of the project defined in Objective (section 1.2).

The four objective defined in this project was achieved. In this project, it have successfully highlighted the weakness and strength of automation tool and for their it progress to comparing component available in the market and the outcome was WebDriver is a suitable component use in this project. Together in the project, it also includes a workable solution with GUI to allow software tester to input test scenario for regression. On top of that, the project also includes a section to clearly identify the area for future expansion which could further improve the system.

Area of Future Expansion

Once a product is release available for public, it does not mean it will be end of development cycle; instead it is the beginning of cycle. The intented research is carries in a control environment and specific objective, and need to complete by agreeable timeline. Due to continuous technologies changes, these researches need to be enhanced in future by other researches who sharing the same interest in this area of work. The following are the suggested improvement that can be taking consideration in making the product better.

Scheduler

Scheduling can be very useful implementation idea. Imagine that, in large organization where interaction with a complex environment and each different host system have its availabilities, and in order to perform cut-off period scenario. Software tester is requirement to wake-up in weird hours and start testing the application.

With this feature, it is capable of telling the application to executed which test collection at which time.

Simple Reporting

WAT currently does not provide any feature to generate report in any format, however software tester have the flexibility to access the repository to extract out the result manually. Reporting can be a subjective argument because of different audience might want different type of presentation. For future enhancement, WAT will include report generating feature to provided testing summaries, at lease a basic version.

Element Selector

Creating records for "Sub-Scenario" can be a bit tricking and required fair bit of technical skill on HTML. As for current, software tester can use 3rd party tool to verify the element name and use it records creation. However, a feature such as Selenium IDE (Alejandro Brizuela, 2012) could be able to speed-up the creation of the sub-scenario.

Multiple Task Execution

To allow multiple Test Collection to be executed at a single period of time, this will eventually improve the productivities providing and reduce the needed of installing application to multiple desktop or laptop if multiple execution is needed.

Client Script Interaction

As mention in Objective (section 1.2), this project only cover validation for server site, and client site validation such as VBScript and JavaScript is not supported at this moment, and will be enhance in future development.

Proxy Setting

Proxy configuration is available in most of the modern web browser, the purposes of proxy is to protect the client request to an authenticate web request, and also assisting in blocking unauthorized outgoing access. In future enhancement, the proxy setting will be embedded into the WebDriver during the object initialization.

Personal evaluation

The idea of taking researching automation started back when customer required us to develop enhance a sub-function without given us sufficient time to perform testing on modules which are link to it. It was a 50/50 chance that it will break the existing functionality; however it was lucky that nothing serious was happen.

When doing the research, I have learn that the automation have a bright future in improving the software qualities. However, the integration to the open source component required a fair bit of time in research, but the result is worth of testing.

Conclusion

As a conclusion to the project, it can be stated that the proposed objective have been successfully achieved. In addition to these, personal development of knowledge and skills for working with .Net and development technology has also been improved.