Documentation
Installation
ATP has been developed in Java. Therefore, a JVM must be installed in your system. In addition, some other components should be present in your computer to run ATP properly, namely:- Browser. The navigation performed by ATP is done by means of a real browser. All in all, a browser
should be installed. This navigation is based on Selenium RC. The browsers (and versions) supported
are the following:
- Windows: Explorer - Firefox - Opera - Chrome
- Linux: Firefox - Opera
- Mac OS X: Firefox - Opera - Safari
- Ptyhon. If you are going to perform security analysis, python should be present in your system. Concretely, the executable file/script called "python" should be accessible from the shell.
- Perl. If you are going to perform usability analysis, perl should be present in your system. Concretely, the executable file/script called "perl" should be accessible from the shell.
Inputs
ATP performs the automated functional testing browsing the different paths of the navigation of a web application. These paths are found by using a novel approach based on the Chinese Postman Problem. During this automated navigation, automated performance and security testing (i.e. dynamic) can be performed in each state of the navigation. In addition, (static) analysis such as compatibility, usability and accessibility can be also performed in the states of the found paths. All in all, the correct navigation structure should be defined first. ATP accepts three kinds of input, namely:- HTML Scripts: These scripts are recorded by using Selenium IDE against the web under test and exported to HTML.
This kind of automation is the Record&Playback (R&P) approach. ATP understands a subset of the Selenium IDE commands, namely:
Type Command Description Mouse buttons click Click in an HTML element. doubleClick Double click in an HTML element. clickAt Click in a position X,Y. Change of state clickAndWait Click and wait to load a web page. doubleClickAndWait Double click and wait to load a page. clickAtAndWait Click in a position and wait to load. Mouse movements mouseDown The user depresses the mouse button. mouseMove Mouse pointer moves inside an element. mouseOut Mouse pointer leaves an element. mouseOver Mouse pointer enters on an element. mouseUp Mouse pointer is over on an element, and the mouse button is released. Keyboard actions type Sets the value of an input field. select Select an option from a drop-down. keyDown Pressing a key (without releasing it yet). keyPress Pressing and releasing a key. keyUp Releasing a key. Assertions assertText Assert that an element text is present. assertNotText Assert that an element text is not present. assertTextPresent Assert that a string is present. assertTextNotPresent Assert that a string is not present. assertValue Assert that an element value is present. assertNotValue Assert that an element value is not present.
- UML 2.0 Models: Such diagrams can be used for describing requirements and driving the test automation process.
This kind of automation is known as Model-Based.
The UML diagrams required by ATP are the following:
- Use case diagrams offer a perspective of the functional requirements of the application interaction with the actors.
- Activity diagrams are used to show details of each use case. These diagrams describe the flow (i.e. the navigation) within a use case.
- Presentation diagrams are used to model the data handled by the web application.
UML 2.0 does not provide a standard way to describe presentation. For that reason, ATP uses UML models based on NDT, which is a methodological approach to model web applications. This diagrams as input created in Enterprise Architect and exported to XMI 2.1. Nevertheless, not only UML diagrams based in NDT can be used as input, in addition any XMI file with the required diagrams (use case, activity, presentation) can be used as input.
- XML Models: ATP proposes a self-defined XSD schema to model the navigation.
These kinds of files are richer than UML in the sense of these files can contain test data and oracles.
In addition, XML models enhaces HTML Scripts since each path can be contemplated (and HTML scripts are alwawys linear).
A XML model for ATP defines a website as a collection of states and transitions. The initial state is always unique, since it identifies the entry point to the navigation. In addition, there is a finite number or web states connected by transitions, which represents the XSD type for a web site. There is a mandatory XML attribute in the definition of a web site named base. This attribute is the starting URL for the navigation. The automation of the browsing will be carried out from this URL.
Each state is recognised by a unique identifier. Moreover, each state can contain a set of data fields and oracles. Each data field contains the following information:
- Locator: Data field identifier.
- Ref: Optional reference to transition (attribute id in transitions).
- Type: Data type. It corresponds to the HTML input types, i.e. text, password, checkbox, radio, submit, reset, file, hidden, image, button.
- Required: Boolean value than indicates whether or not the data field is mandatory.
- Stereotype: One of the following types: email, date, name, surname, address, string, integer.
- Value: Collection of values of the data field.
Each oracle (assertion) contains the following information:
- Locator: Oracle identifier.
- Ref: Optional reference to transition (attribute id in transitions).
- Type: Oracle type. It could be: text (assertion for a text in the element identified bylocator), notText (the opposite of text), textPresent (assertion for a text present in the any element in the web page), textNotPresent (the opposite of textPresent), value (assertion for a value in the element identified by locator), notValue (the opposite of value).
Finally, web transitions are composed by an attribute called from (which is the identifier of the web page source) and a collection of actions and web targets (attribute to). The action attribute is composed by the following fields:
- Target: Identifier (id field) of the destination web page.
- Event: Literal that describes the action performed. This field follows the same notation
described in the following table:
Command Description click Click in an HTML element. dbclick Double click in an HTML element. mousedown A mouse button is pressed mousemove The mouse is moved mouseout The mouse is moved off an element. mouseover Mouse pointer enters on an element. mouseup The mouse is moved over an element. keydown A keyboard key is pressed or held down. keypress A keyboard key is pressed. keyup A keyboard key is released.
- Key: Optional field containing the button that triggers events keypress, keydown and keyup.
Each transition changes the navigation from a source state to another target. This destination is defined in the attribute to, which has the following properties:
- State: Target state identification (id).
- Id: Optional identification for the transition. This id is used for condition data fields and oracles tagged with the attribute ref.
- Weight: Integer value which determines the importance of the transition. By default, each transition is assigned a numeric weight of 5. When the CPP algorithm is applied, it looks for this weight to decide which the following transition in the navigation is. If there is several alternatives, the transition with a higher weight will be selected first.
Transitions
Transtions in ATP are understood as a set of user actions resulting in a change of web state. The simpler kind of transition is the action of clicking a web link to change from onw page to another. The guard in the transitions (i.e., the identifier of the transition) is translated by ATP following this procedure:
Function LookForHTMLElement(Guard)
Found = nothing
For each frame in the frameset (if frames exist)
For each HTML element in the frame
Found = Look for Guard in the id/name/title/value/alt tag
If Not Found
Found = Look for Guard as text
If Not Found
Found = Execute Guard as XPath expression
End If
End If
End For
End For
Return Found
End Function
Test Data and Oracles
ATP uses the data (input to test cases) and oracles (expected outcome) whitin the HTML Scripts and XML Files. This information is stored in separated Excel Spreadsheet per navigation path. This approach is kwnon as Data-Driven. Test data is separated from Oracle by an enmpty column in the spreadsheets. New test cases can be achieved by adding new rows (data and oracles) in the data files.Commands
ATP has been developed as a shell-tool. The ATP commands are the following:> atp create
To create all the possible tests cases.> atp run
To execute and also reports the previously created test cases> atp report
To show the generated report after running test cases> atp clean
To delete all the previously created tests cases, libraries, and reports> atp list
To show the ATP internal configuration parameters> atp set [parameter] [value]
To change those configuration parameters> atp help [command]
To show some help about the configuration parameters> atp data
To open the generated spreadsheets (which store the test data and oracles)> atp data add
To add new test (random) data rows the generated spreadsheetsConfiguration
In order to configure ATP, you should change the following parameters (in atp.properties):- The following three commands are mandatory to be configured before ATP execution:
root=Output folder, where test cases will be generated using [atp create]
navigation_dir=Folder where navigation structure (XMI, XML, or HTML) is stored.
navigation_type=Type of input stored in [navigation_dir] (XMI, XML, or HTML).
- The following commands are not mandatory but are very important since they indicates which king of testing (compatibility, security) and analysis (compatibility, usability, accessibility) will be performed.
Functional testing (i.e. automation of the navigation) is always performed. In addition, the parameter
source
indicates whether or not the Java classes -wrappers of the integrated testing/analysis tools- are visible:
compatibility=Compatibility analysis flag (true|false).
performance=Performance testing flag (true|false).
accessibility=Accessibility analysis flag (true|false).
security=Security testing flag (true|false).
usability=Usability testing flag (true|false).
source=Boolean value (true|false), it indicates whether or not the source code of Java test cases is generated.
- The following commands tune the features of the different testing (functionality, performance, security) and analysis (compatibility, usability, accessibility) aspectes:
seleniumPort=Selenium server port (for the automated navigation).
browser=Browser used in the automated navigation (firefox|iexplore|googlechrome|safari|opera)
timeout=Time to wait the load of each web state in the automated navigation.
speed=Delay (in milliseconds) for each Selenium operation.
cssProfile=CSS profile for compatibility analysis: css2|css21|css3|svg|svgbasic|svgtiny|atsc-tv|mobile|tv
htmlProfile=HTML profile for compatibility analysis: html401strict|html401transitional|xhtml401strict|xhtml401frameset
jmeterPort=Port for JMeter server (performance testing).
concurrentUsers=Number of virtual concurrent users for performance testing.
maxResponseTime=Maximum response time (in milliseconds) per web state for performance testing.
minThroughput=Minimum throughput (in samples/s) per web state for performance testing.
minBitrate=Minimum bitrate (in KB/sec) per web state for performance testing.
guidelines=Accessibility guidelines, one of the following values:BITV1|508|STANCA|WCAG1-A|WCAG1-AA|WCAG1-AAA|WCAG2-A|WCAG2-AA|WCAG2-AA
acheckerId=Identifier to interact with AChecker
acheckerUrl=URL for AChecker. By default http://atp.vv.si/checkacc.php
vulnerabilities=Vulnerability databases (separated with commas). Possibilities: crlf,exec,file,sql,xss,backup,htaccess,blindsql,permanentxss,nikto.
method=HTTP method for security testing attacks (get,post).
usabilitMethods=Usability methods. Possibilities: form,maintain,navigate,read,headinfo,bodyinfo.