Testatoo

1. Introduction


Testatoo is a web user interface testing tool. It’s the result of numerous real-world observations of developers in the trenches in the area of GUI testing. Working for many years to promote the TDD/BDD approaches, we often faced difficulties in their implementation for the graphical layer of applications.

The "TEST FIRST" principle excludes all scenario recorder based approaches that only allow you to write "at posteriori" tests. Our experience has taught us that this path is a dead-end (but we reserve this for another discussion).

Another problem is UI tests are brittle and costly! We do think that this is due to the lack of abstraction in existing UI testing tools.

Testatoo provides, on one hand, an abstraction of the UI business domain through an expressive API and, on the other hand, a way to express this domain via a DSL (a button semantically stays a button, whatever the technology). With Testatoo, you can therefore write tests with a seldom achieved level of expressiveness and make those tests INDEPENDENT of the underlying technology.

Testatoo can therefore transform the tests in real assets, present throughout the life of the application and always in tune with the latest version of the application.

Testatoo is an opinionated tool. It forces the user to see and use the domain layer as a composition and aggregation of UI components (in Testatoo, everything is a UI component).

Testatoo is a library to write functional tests. Functional tests means SPECIFICATIONS.


As a SPECIFIER, I want to express my test (intention) BEFORE UI is coded. So, if the intention is to have a page with two radio buttons to select the gender (male or female), the test can be expressed like this :

The male_radio should be unchecked
and it should have label "Male".

The female_radio should be unchecked
and it should have label "Female".

Check the male_radio:
the male_radio should be checked,
the female_radio should be unchecked.

Check the female_radio:
the female_radio should be checked,
the male_radio should be unchecked.

The Testatoo syntax to express this specification is:

male_radio.should {
    be unchecked
    have label('Male')
}

female_radio.should {
    be unchecked
    have label('Female')
}

check male_radio
male_radio.should { be checked }
female_radio.should { be unchecked }

check female_radio
male_radio.should { be unchecked }
female_radio.should { be checked }

So we can see that Testatoo is close to the natural language.

2. Design and architecture


diag 196ed5835f53aac566ca86b1cb0eaf35

Testatoo is built on top of Selenium, it can work with any browser supported by Selenium (Edge, Chrome, Firefox, Safari). Even though Testatoo provides an extra layer of convenience and productivity, it is always possible to "fallback" to the WebDriver level to do something directly, if you need to.

Testatoo adds a powerful DSL, while maintaining the usage of Groovy and keeps the advantages of a strongly typed language.

3. Quick Start

3.1. Installation with Maven


These are the dependencies to include if you want to integrate testatoo in your Maven installation file. The following installation if for JUnit. For another test runner, adapt the first dependency.

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.testatoo</groupId>
   <artifactId>testatoo</artifactId>
   <version>2.0.b26</version>
   <scope>test</scope>
</dependency>

3.2. A first UI test example with JUnit


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package doc.junit

import org.junit.AfterClass
import org.junit.BeforeClass
import org.junit.Test
import org.junit.experimental.categories.Category
import org.junit.runner.RunWith
import org.junit.runners.JUnit4
import org.openqa.selenium.chrome.ChromeDriver
import org.testatoo.bundle.html5.input.InputTypeText
import org.testatoo.category.UserAgent
import org.testatoo.core.component.field.TextField
import org.testatoo.evaluator.webdriver.WebDriverEvaluator

import static org.testatoo.core.Testatoo.*

@RunWith(JUnit4)
@Category(UserAgent.Chrome)
class JunitStarterTest {

    @BeforeClass
    static void setup() {
        System.setProperty('webdriver.chrome.driver', '/usr/bin/chromedriver')
        config.evaluator = new WebDriverEvaluator(new ChromeDriver()) (1)

        visit 'http://www.google.ca' (2)
    }

    @Test
    void google_search_field_should_be_visible() {
        // Write you test here
        TextField search = $('#lst-ib') as InputTypeText    (3)
        search.should {
            have focus
            be visible
        }
    }

    @AfterClass
    static void tearDown() {
        config.evaluator.close() (4)
    }
}
1 The browser is launched through a Webdriver Wraper (WebDriverEvaluator).
2 Visit the page to be tested: Testatoo is ready.
3 Write you first test ($ usage is explained after)
4 At the end of the test don’t forget to close the browser.

4. Philosophy

Testatoo is a strongly opinionated tool. There is no anonymous "WebElement". Everything needs to be a COMPONENT. Testatoo is a typed UI component tool.

In the previous sample the line:

TextField search = $('#lst-ib') as InputTypeText

expressed that we try to find an HTML5 input of type text with id: 'lst-ib' ⇒ <input type="text"/> but we want to use it as a formal TextField

The associated import is :

import org.testatoo.core.component.field.TextField
import org.testatoo.bundle.html5.input.InputTypeText
The challenge it’s to never have import on a bundle (everything int the package org.testatoo.bundle) but as possible only import from core (org.testatoo.core.component.field.TextField). The core package is like an Universal UI Domain definition.

Testatoo provides by default the HTML5 bundle to interact with basic HTML5 components

5. Compatibility

Testatoo is based on Selenium 3 and definitely turned to the future. But for the moment all the vendor driver are not at the same level and it’s an on going process. We take care to be 100% compatible with all browser to offer you a full abstraction of Selenium. But for the moment it’s not the case

5.1. Chrome

Fully compatible. We strongly encourage to use it as your default test Browser.

5.2. Firefox

Not fully

  1. Selenium Actions are not implemented see: https://github.com/mozilla/geckodriver/issues/233

  2. FF don’t have the focus on the browser when page is opened: https://github.com/mozilla/geckodriver/issues/394

5.3. Edge

5.4. Safari

5.5. Fallback

In any case you can fallback to native webdriver like this:

WebDriver webdriver = config.evaluator.driver

6. Content interaction


6.1. The Browser

The Browser + Navigation

6.2. Component selection

6.2.1. The $ function

The $ function is the access point for fetching component on the page. The $ function accepts a CSS3 selector to target your component. The signature of the $ function is the following:

$('css selector')

// Samples
Button button = $('#button') as Button              (1)
Button reset = $('.btn-secondary') as Button        (2)
Button submit = $('input:eq(1)') as Button          (3)
Button myButton = $('[data-role=myRole]') as Button (4)
1 Matches the DOM element on the page with id: 'button'
2 Matches the DOM element on the class: 'btn-secondary'
3 Matches the DOM element on the page that is the second button
4 Matches the DOM element on the page that has attribute "data-role" equals to "myRole"

Our experience demonstrated that we never need complex selector. If you write tests first, we use the semantic carried by the tested component. In the code, this is reflected by a custom tag, the sequence in the page, a role attribute, or an id. In all these cases, a css3 selector is more than enough.

Once the component is selected, it MUST BE TYPED through the "as" keyword.

TextField textField = $('#text_field') as InputTypeText

In Testatoo you never select an anonymous DOM element. Every DOM element must be the representation of a UI concept. But wait a moment: How can you know if a button is a button!? Indeed, in HTML, a Button can be represented in many ways:

<button id="button">My Button</button>
<input id="button" type="button" value="My Button">
<input id="button" type="submit" value="My Button">

This is the magic of Testatoo. We will see later how it works and how to override this behavior and easily add new component types.

The $ function NEEDS to target a UNIQUE component. If you want to work with a list of components you need to use the $$ function.

The following code fails because through the selector expression, we try to target a list of items (options) of a select to evaluate their visibility.

Item item = $('select option') as Item;
assert item.visible()

6.2.2. The $$ function

TODO

6.2.3. The find function

TODO

6.3. Events on components

6.3.1. Mouse events

Supported mouse events

click_on

double_click_on

right_click_on

hovering_mouse_on

drag …​ on …​

Button button = $('#button') as Button
Panel panel_1 = $('#panel_1') as Panel
Panel panel_2 = $('#panel_2') as Panel

click_on button
double_click_on button
right_click_on button
hovering_mouse_on panel_1
drag panel_1 on panel_2

6.3.2. Keyboard events

For the keyboard events we have done a nice work to make this easy:

Only ONE Method to represent them all

the type(…​) method

TextField textField = $('#textfield') as TextField

click_on textField
type('testatoo')

If you need the usage of a simple key modifier…​.

TextField textField = $('#textfield') as TextField

click_on textField
type(SHIFT + 'testatoo') // => TESTATOO

Or a complex one…​.

TextField textField = $('#textfield') as TextField

click_on textField
type(CTRL + ALT + SHIFT + 'x')

6.3.3. Touch events

Testatoo does not support touch event for the moment.

7. Write a test - Testatoo language


TODO

8. Write a test - BDD style


TODO

9. Components reference

9.1. Component

This is the base for all others components.

States

enabled

disabled

available

missing

hidden

visible

contains

displays

component.should {
    be available
    be enabled
    be visible
}

component = $('#submit') as Component
component.should { be disabled }

component = $('#hidden_panel') as Component
component.should { be hidden }

component = $('#non_existing_id') as Component
component.should { be missing }

All Testatoo components inherit these states.

9.2. Button

Button
a button refers to any graphical control element that provides the user a simple way to trigger an event, like searching for a query at a search engine, or to interact with dialog boxes, like confirming an action.
— Wikipedia definition
States Properties Intentions

-

text

-

button.should { have text('My Button Text') }

9.3. CheckBox

CheckBox
CheckBox
a checkbox is a small box that, when selected by the user, shows that a particular feature has been enabled or a particular option chosen. Check boxes are commonly used when more than one option may need to be selected.
— inspired from Oxford dictionaries definition
States Properties Intentions

checked

label

check

unchecked

uncheck

checkbox.should {
    have label('Check me out')
    be unchecked
}

check checkbox
checkbox.should { be checked }

uncheck checkbox
checkbox.should { be unchecked }

9.4. Radio

Radio
a radio button allows the user to choose only one of a predefined set of options. Radio buttons are arranged in groups of two or more and displayed on screen as, for example, a list of circular holes that can contain white space (for unselected) or a dot (for selected). Each radio button is normally accompanied by a label describing the choice that the radio button represents. The choices are mutually exclusive; when the user selects a radio button, any previously selected radio button in the same group becomes deselected (making it so only one can be selected).
— inspired from Wikipedia definition
States Properties Intentions

checked

label

check

unchecked

checked_radio.should {
    have label('Radio checked')
    be checked
}

unchecked_radio.should {
    have label('Radio unchecked')
    be unchecked
}

check unchecked_radio

unchecked_radio.should { be checked }
checked_radio.should { be unchecked }

9.5. DropDown

DropDown
a dropdown allows the user to choose one value from a list. When a dropdown is inactive, it displays a single value. When activated, it displays (drops down) a list of values, that can be eventually grouped by theme, from which the user may select one. When the user selects a new value, the control reverts to its inactive state, displaying the selected value.
— inspired from Wikipedia definition
States Properties Intentions

-

label

select

number of items

items

number of groups

groups

os_list.should {
    have label('OS')
    have 8.items
    have selectedItem('None')
    have items('None', 'Ubuntu', 'Fedora', 'Gentoo', 'XP', 'Vista', 'FreeBSD', 'OpenBSD')
    have 3.groups
    have groups('linux', 'win32', 'BSD')
}

on os_list select 'Ubuntu'
os_list.should { have selectedItem('Ubuntu') }

9.6. Group

Group is used for groups of items in dropDowns.

DropDown
States Properties Intentions

-

value

-

items

Group linux_group = os_list.group('linux') // Or os_list.groups[0]

linux_group.should {
    have value('linux')
    have items('Ubuntu', 'Fedora', 'Gentoo')
}

9.7. ListBox

ListBox ListBox

The second image show all the items available in the ListBox.

a listBox allows the user to select one or more items from a list contained within a static, multiple line text box. The user clicks inside the box on an item to select it, sometimes in combination with the ⇧ Shift or Ctrl in order to make multiple selections. "Control-clicking" an item that has already been selected, unselects it.
— Wikipedia definition
States Properties Intentions

label

select

number of items

items

selectedItems

number of visible items

cities.should {
    have label('Cities list')
    have 6.items
    have items('Montreal', 'Quebec', 'Montpellier', 'New York', 'Casablanca', 'Munich')
    have selectedItems('Montreal')

    have 3.visibleItems     // See the first image at the begin of this section
}

on cities select 'Montpellier', 'New York'
cities.should { have selectedItems('Montreal', 'Montpellier', 'New York') }

// Or
Item montreal = cities.item('Montreal')
Item montpellier = cities.item('Montpellier')
Item ny = cities.item('New York')
cities.should { have selectedItems(montreal, montpellier, ny) }

9.8. Item

Item is used for dropDowns and listBoxes.

DropDown ListBox

States Properties Intentions

selected

value

select

unselected

unselect

Item os = os_list.item('Gentoo')  // Or os_list.items[1]
os.should {
    have value('Gentoo')
    be unselected
}

select os
os.should { be selected }

ListBox cities = $('#cities') as MultiSelect
Item city = cities.item('Montpellier')
city.should {
    have value('Montpellier')
    be unselected
}

select city
city.should { be selected }

9.9. Field

This is the base component for all types of fields.

States Properties Intentions

empty

text

fill

filled

label

clear

readOnly

placeholder

required

value

optional

valid

invalid

All following types of fields inherit these states, properties and intentions.

9.9.1. TextField

TextField
States Properties Intentions

length

a textfield allows the user to input text information to be used by the program. It can a single-line text box when only one line of input is required, and a multi-line text box if more than one line of input may be required.
— inspired from Wikipedia definition
textfield.should {
    be empty
    be required
    have placeholder('Placeholder')
    have length(20)
}

fill textfield with '1234'

textfield.should {
    be filled
    have value('1234')
}

9.9.2. ColorField

ColorField

Definition: a colorField is used for fields that should contain a color.

9.9.3. DateTimeField

DateTimeField

Definition: a dateTimeField allows the user to select a date and time (with time zone).

9.9.4. DateField

DateField-1
DateField-2
DateField-3

Definition: a dateField allows the user to select a date.

States Properties Intentions

inRange

maximum

step

outOfRange

minimum

date.should {
    be inRange
    have value('')
    have step(0)
    have maximum('2012-06-25')
    have minimum('2011-08-13')
}

set date to '2011-06-26'
date.should { have value('2011-06-26') }

9.9.5. TimeField

TimeField-1
TimeField-2

Definition: a timeField allows the user to select a time (no time zone).

9.9.6. MonthField

MonthField-1
MonthField-2

Definition: a monthField allows the user to select a month and year.

9.9.7. WeekField

WeekField-1
WeekField-2

Definition: a weekField allows the user to select a week and year.

9.9.8. EmailField

EmailField
States Properties Intentions

length

Definition: an emailField is used for fields that should contain an e-mail address.

9.9.9. PasswordField

States Properties Intentions

length

PasswordField

Definition: a passwordField is used for fields that should contain a password.

9.9.10. NumberField

NumberField

Definition: a numberField is used for fields that should contain a numeric value.

States Properties Intentions

inRange

maximum

step

outOfRange

minimum

number.should {
    have minimum(0)
    have maximum(64)
    have step(8)
    have value(2)
    be inRange
}

fill number with 150
number.should { be outOfRange }

9.9.11. RangeField

RangeField

Definition: a rangeField is used for fields that should contain a value within a range.

States Properties Intentions

inRange

maximum

step

outOfRange

minimum

range.should {
    have minimum(0)
    have maximum(50)
    have value(10)
    have step(5)
    is inRange
}

set range to 40

range.should { have value(40) }

9.9.12. PhoneField

PhoneField

Definition: a phoneField is used for fields that should contain a telephone number.

9.9.13. SearchField

SearchField
States Properties Intentions

length

Definition: a searchField is used for search fields (a search field behaves like a regular text field).

9.9.14. URLField

URLField

Definition: an URLField is used for fields that should contain a URL address.

9.10. Form

Form

Definition: a form allows the user to collect user input. Form elements are different types of input elements, checkboxes, radio buttons, submit buttons, and more.

States Properties Intentions

valid

-

reset

invalid

-

submit

email_field.should {
    have focus
    be focused // Or
    be optional

}
password_field.should { be required }

form.should {
    contain(email_field, password_field)
    be invalid
}

fill email_field with 'invalid_email'
fill password_field with 'a password'

form.should { be invalid }

fill email_field with 'valid@email.org'

form.should { be valid }

9.11. Heading

Heading

Definition: a heading allows the user to put a title for a page or a paragraph.

States Properties Intentions

-

text

-

first_heading.should { have text('heading 1') }
last_heading.should { have text('heading 6') }

9.12. Image

Image

Definition: an image allows the user to insert an image (gif, jpg, png,…​) from a source file

States Properties Intentions

-

reference

-

image.should { have reference(BASE_URL + 'img/Montpellier.jpg') }
Link-1
Link-2

Definition: a link allows the user to navigate from page to page.

States Properties Intentions

-

text

-

-

reference

-

link.should {
    have text('Link to dsl page')
    have reference(BASE_URL + 'dsl.html')
}

9.14. ListView

ListView-1
ListView-2

Definition: a listView allows the user to display items in a list format (with bullet points or numbers, indentation,…​).

States Properties Intentions

items

listview.should {
    have 5.items
    have items('Item 1', 'Item 2', 'Item 3', 'Item 4', 'Item 4')
}

9.15. Panel

Panel
a panel is a particular arrangement of information grouped together for presentation to the user.
— inspired from Wikipedia definition
States Properties Intentions

-

title

-

panel.should { have title('My panel title') }

9.16. Datagrid

Definition: a datagrid is a component that allows user to present data in a tabular view, with cells organized in rows and columns.

DataGrid
States Properties Intentions

-

rows

-

-

columns

-

datagrid.should {
    have 4.columns
    have columns('', 'Column 1 title', 'Column 2 title', 'Column 3 title')

    have 4.rows
    have rows('Row 1', 'Row 2', 'Row 3', 'Row 4')
}

9.16.1. Column

Definition: a column is composed of cells and could have a title.

States Properties Intentions

-

cells

-

-

title

-

Column column = datagrid.column('Column 2 title') // Or datagrid.columns[2]
column.should {
    have title('Column 2 title')

    have 4.cells
    have cells('cell 12', 'cell 22', 'cell 32', 'cell 42')
}

9.16.2. Row

Definition: a row is composed of cells and could have a title.

States Properties Intentions

-

cells

-

-

title

-

Row row = datagrid.row('Row 3') // Or datagrid.row[2]
row.should {
    have title('Row 3')

    have 3.cells
    have cells('cell 31', 'cell 32', 'cell 33')
}

9.16.3. Cell

Definition: a cell is the finest element of a datagrid.

States Properties Intentions

-

value

-

datagrid.rows()[2].cells()[1].should {
    have value('cell 32')
}

10. The HTML5 bundle


Testatoo comes with a bunch of predefined HTML components. These components are not only the standard representation of HTML elements, but also a powerful base to build your own components. Each component supports States and Properties. One of the strengths of Testatoo is the ability to add support for new States and Properties and to override existing States and Properties, on existing or new components.

11. Create a new component


TODO

12. Create a new bundle


TODO