Documentation

Installation

ATP has been developed in Java. Therefore, a JVM must be installed in your system. In addition, some other components should be present in your computer to run ATP properly, namely:

Browser. The navigation performed by ATP is done by means of a real browser. All in all, a browser should be installed. This navigation is based on Selenium RC. The browsers (and versions) supported are the following:
- Windows: Explorer - Firefox - Opera - Chrome
- Linux: Firefox - Opera
- Mac OS X: Firefox - Opera - Safari
Ptyhon. If you are going to perform security analysis, python should be present in your system. Concretely, the executable file/script called "python" should be accessible from the shell.
Perl. If you are going to perform usability analysis, perl should be present in your system. Concretely, the executable file/script called "perl" should be accessible from the shell.

Inputs

ATP performs the automated functional testing browsing the different paths of the navigation of a web application. These paths are found by using a novel approach based on the Chinese Postman Problem. During this automated navigation, automated performance and security testing (i.e. dynamic) can be performed in each state of the navigation. In addition, (static) analysis such as compatibility, usability and accessibility can be also performed in the states of the found paths. All in all, the correct navigation structure should be defined first. ATP accepts three kinds of input, namely:

HTML Scripts: These scripts are recorded by using Selenium IDE against the web under test and exported to HTML. This kind of automation is the Record&Playback (R&P) approach. ATP understands a subset of the Selenium IDE commands, namely:

Type	Command	Description
Mouse buttons	click	Click in an HTML element.
	doubleClick	Double click in an HTML element.
	clickAt	Click in a position X,Y.
Change of state	clickAndWait	Click and wait to load a web page.
	doubleClickAndWait	Double click and wait to load a page.
	clickAtAndWait	Click in a position and wait to load.
Mouse movements	mouseDown	The user depresses the mouse button.
	mouseMove	Mouse pointer moves inside an element.
	mouseOut	Mouse pointer leaves an element.
	mouseOver	Mouse pointer enters on an element.
	mouseUp	Mouse pointer is over on an element, and the mouse button is released.
Keyboard actions	type	Sets the value of an input field.
	select	Select an option from a drop-down.
	keyDown	Pressing a key (without releasing it yet).
	keyPress	Pressing and releasing a key.
	keyUp	Releasing a key.
Assertions	assertText	Assert that an element text is present.
	assertNotText	Assert that an element text is not present.
	assertTextPresent	Assert that a string is present.
	assertTextNotPresent	Assert that a string is not present.
	assertValue	Assert that an element value is present.
	assertNotValue	Assert that an element value is not present.

UML 2.0 Models: Such diagrams can be used for describing requirements and driving the test automation process. This kind of automation is known as Model-Based. The UML diagrams required by ATP are the following:
- Use case diagrams offer a perspective of the functional requirements of the application interaction with the actors.
- Activity diagrams are used to show details of each use case. These diagrams describe the flow (i.e. the navigation) within a use case.
- Presentation diagrams are used to model the data handled by the web application.
UML 2.0 does not provide a standard way to describe presentation. For that reason, ATP uses UML models based on NDT, which is a methodological approach to model web applications. This diagrams as input created in Enterprise Architect and exported to XMI 2.1. Nevertheless, not only UML diagrams based in NDT can be used as input, in addition any XMI file with the required diagrams (use case, activity, presentation) can be used as input.

XML Models: ATP proposes a self-defined XSD schema to model the navigation. These kinds of files are richer than UML in the sense of these files can contain test data and oracles. In addition, XML models enhaces HTML Scripts since each path can be contemplated (and HTML scripts are alwawys linear).

A XML model for ATP defines a website as a collection of states and transitions. The initial state is always unique, since it identifies the entry point to the navigation. In addition, there is a finite number or web states connected by transitions, which represents the XSD type for a web site. There is a mandatory XML attribute in the definition of a web site named base. This attribute is the starting URL for the navigation. The automation of the browsing will be carried out from this URL.

Each state is recognised by a unique identifier. Moreover, each state can contain a set of data fields and oracles. Each data field contains the following information:

Locator: Data field identifier.
Ref: Optional reference to transition (attribute id in transitions).
Type: Data type. It corresponds to the HTML input types, i.e. text, password, checkbox, radio, submit, reset, file, hidden, image, button.
Required: Boolean value than indicates whether or not the data field is mandatory.
Stereotype: One of the following types: email, date, name, surname, address, string, integer.
Value: Collection of values of the data field.

Each oracle (assertion) contains the following information:

Locator: Oracle identifier.
Ref: Optional reference to transition (attribute id in transitions).
Type: Oracle type. It could be: text (assertion for a text in the element identified bylocator), notText (the opposite of text), textPresent (assertion for a text present in the any element in the web page), textNotPresent (the opposite of textPresent), value (assertion for a value in the element identified by locator), notValue (the opposite of value).

Finally, web transitions are composed by an attribute called from (which is the identifier of the web page source) and a collection of actions and web targets (attribute to). The action attribute is composed by the following fields:

Target: Identifier (id field) of the destination web page.

Event: Literal that describes the action performed. This field follows the same notation described in the following table:

Command	Description
click	Click in an HTML element.
dbclick	Double click in an HTML element.
mousedown	A mouse button is pressed
mousemove	The mouse is moved
mouseout	The mouse is moved off an element.
mouseover	Mouse pointer enters on an element.
mouseup	The mouse is moved over an element.
keydown	A keyboard key is pressed or held down.
keypress	A keyboard key is pressed.
keyup	A keyboard key is released.

Key: Optional field containing the button that triggers events keypress, keydown and keyup.

Each transition changes the navigation from a source state to another target. This destination is defined in the attribute to, which has the following properties:

State: Target state identification (id).
Id: Optional identification for the transition. This id is used for condition data fields and oracles tagged with the attribute ref.
Weight: Integer value which determines the importance of the transition. By default, each transition is assigned a numeric weight of 5. When the CPP algorithm is applied, it looks for this weight to decide which the following transition in the navigation is. If there is several alternatives, the transition with a higher weight will be selected first.

Transitions

Transtions in ATP are understood as a set of user actions resulting in a change of web state. The simpler kind of transition is the action of clicking a web link to change from onw page to another. The guard in the transitions (i.e., the identifier of the transition) is translated by ATP following this procedure:

Function LookForHTMLElement(Guard)
  Found = nothing
  For each frame in the frameset (if frames exist)
    For each HTML element in the frame
      Found = Look for Guard in the id/name/title/value/alt tag
      If Not Found
        Found = Look for Guard as text
        If Not Found
          Found = Execute Guard as XPath expression
        End If
      End If
    End For
  End For
  Return Found
End Function

Test Data and Oracles

ATP uses the data (input to test cases) and oracles (expected outcome) whitin the HTML Scripts and XML Files. This information is stored in separated Excel Spreadsheet per navigation path. This approach is kwnon as Data-Driven. Test data is separated from Oracle by an enmpty column in the spreadsheets. New test cases can be achieved by adding new rows (data and oracles) in the data files.

Commands

ATP has been developed as a shell-tool. The ATP commands are the following:

> atp create

To create all the possible tests cases.

> atp run

To execute and also reports the previously created test cases

> atp report

To show the generated report after running test cases

> atp clean

To delete all the previously created tests cases, libraries, and reports

> atp list

To show the ATP internal configuration parameters

> atp set [parameter] [value]

To change those configuration parameters

> atp help [command]

To show some help about the configuration parameters

> atp data

To open the generated spreadsheets (which store the test data and oracles)

> atp data add

To add new test (random) data rows the generated spreadsheets

Configuration

In order to configure ATP, you should change the following parameters (in atp.properties):

The following three commands are mandatory to be configured before ATP execution:

root=Output folder, where test cases will be generated using [atp create]

navigation_dir=Folder where navigation structure (XMI, XML, or HTML) is stored.

navigation_type=Type of input stored in [navigation_dir] (XMI, XML, or HTML).

The following commands are not mandatory but are very important since they indicates which king of testing (compatibility, security) and analysis (compatibility, usability, accessibility) will be performed. Functional testing (i.e. automation of the navigation) is always performed. In addition, the parameter source indicates whether or not the Java classes -wrappers of the integrated testing/analysis tools- are visible:

compatibility=Compatibility analysis flag (true|false).

performance=Performance testing flag (true|false).

accessibility=Accessibility analysis flag (true|false).

security=Security testing flag (true|false).

usability=Usability testing flag (true|false).

source=Boolean value (true|false), it indicates whether or not the source code of Java test cases is generated.

The following commands tune the features of the different testing (functionality, performance, security) and analysis (compatibility, usability, accessibility) aspectes:

seleniumPort=Selenium server port (for the automated navigation).

browser=Browser used in the automated navigation (firefox|iexplore|googlechrome|safari|opera)

timeout=Time to wait the load of each web state in the automated navigation.

speed=Delay (in milliseconds) for each Selenium operation.

cssProfile=CSS profile for compatibility analysis: css2|css21|css3|svg|svgbasic|svgtiny|atsc-tv|mobile|tv

htmlProfile=HTML profile for compatibility analysis: html401strict|html401transitional|xhtml401strict|xhtml401frameset

jmeterPort=Port for JMeter server (performance testing).

concurrentUsers=Number of virtual concurrent users for performance testing.

maxResponseTime=Maximum response time (in milliseconds) per web state for performance testing.

minThroughput=Minimum throughput (in samples/s) per web state for performance testing.

minBitrate=Minimum bitrate (in KB/sec) per web state for performance testing.

acheckerId=Identifier to interact with AChecker

acheckerUrl=URL for AChecker. By default http://atp.vv.si/checkacc.php

vulnerabilities=Vulnerability databases (separated with commas). Possibilities: crlf,exec,file,sql,xss,backup,htaccess,blindsql,permanentxss,nikto.

method=HTTP method for security testing attacks (get,post).

usabilitMethods=Usability methods. Possibilities: form,maintain,navigate,read,headinfo,bodyinfo.