Testing Framework

# Documentation

## Introduction

Fressia it's basically composed of:
• A really simple scripting language - the Fressia language which is a DSL.
• An interpreter to execute scripts written in Fressia language.
• A results report generator.
In the following sections you'll find all what you need to write and execute Fressia scripts.

## Suites

A suite script file is a normal text file that follows this syntax:
suite <suite_name> {

// test definitions
//       or
// suite import definitions
//       or
// variable definitions

}

The suite name is user defined and is not restricted to be unique, but if you have nested suites - using the import sentence - you might have some troubles if you have repeated names among several suites.

The file name can be anything and it doesn't have to be related to the suite name - although, despite is not a restriction, we recommend using the extension 'fs'.

## Tests

A test is composed of the action block and the asserts block. General syntax is:
test <test_name> {
[action]: <the test action definition>;

// parameters
//   or
// events

} asserts {

// asserts

}

The test name is also user defined and it must be unique within the suite it belongs to. The action definition just accepts the reserved words that are explained in the sections below.

Both parameter and event set are related to every specific action - some actions don't need them - and follow the same structure.

## Asserts and Events

Asserts are applied against the action response after it has been executed while Events are executed during the action execution. They:
• Implicitly return a boolean value.
• Are all joined, as explained below, and treated as a single implicit boolean expression. If this final expression is false then the whole test is reported as failed.
• Are not strictly related to actions but every action has an assert (event) set that is allowed to accept.
The syntax of a single assert (event) expression is quite simple. It's specified by two reserved words in this way:

In case of Asserts:
target_type condition ("argument");
• The target type is to indicate the 'data type' that Fressia will use to handle the action response after the action execution.
• The condition is what you actually want to check in the response using the information provided in the argument. The latter is not needed in some cases; so some expressions just look like:
target_type condition;
In case of Events:
event_type event ("parameters");
• The first word is to indicate the 'event type' that Fressia will use to handle the action event execution.
• The second word is the event that is going to be triggered during the action execution using the information provided in the parameters. The latter is not needed in some cases; so some expressions just look like:
event_type event;
They can be combined using these boolean operators:
• and (you can also use &&).
• or (you can also use ||).
• not (you can also use !).
Examples:
not (t c ("a"));
!t1 c1 ("a1") && (t2 c2 ("a2") or t3 c3 ("a3"));

Single assert (event) expressions separated by semicolon are joined with the '&&' operator. Hence, these two expressions are equivalent:
t c ("a");
e c ("b");

t c ("a") and e c ("b");


### Text Asserts

Here, the action response is treated as flat text. They can be:
text contains ("some text");
text equals ("some text");
text matches ("some regular expression");
Regular expressions must be in the Java format.

### XML/HTML Asserts

In this case the assert response is treated as an XML DOM. You can have two of these asserts:
xml isValid;
Above is used to check if the action response is a well-formed XML document.
xml validates ("/path/to/an/XSD/file");
Above is used to check if the action response XML document validates an XSD. Notice that XSD is a rich XML-ish language and, combined with the first XML assert, it allows you to test anything from an XML document.

Besides, you can check if an HTML document is well-formed and W3C compliant with:
html isValid;

### Email Asserts

These are valid within the context of an email action only. Here you don't have an action response but the state of the Fressia internal SMTP server once it has received (or not) a set of emails. So, these asserts are things that you may wonder about that email set. They can be to:

Count the number of messages received by the internal SMTP server:
messages count ("some integer number");

Check if all messages contain a particular text in their body texts:
messages eachBodyContains ("some text");

Check if any message contains a particular text in its body text:
messages eachSubjectContains ("some text");

Check if all messages contain a particular text in their subject texts:
messages anyBodyContains ("some text");

Check if any message contains a particular text in its subject text:
messages anySubjectContains ("some text");

Check if any message sender ('From' header field) contains an email address:
messages anySenderContains("an email address");

Check if all message senders ('From' header field) contain an email address:
messages eachSenderContains ("an email address");

messages anyRecipientContains ("an email address");

messages eachRecipientContains ("an email address");
See an example here.

## Variables

You can define variables within the scope of a suite - variable definitions are *not* allowed within a test scope. The syntax is:
$<name> = <value>; The value type can be: • String - enclosed by quotes. Example: $var = "hola";
• Integer
• Float
• Boolean - true or false
At the moment, combined with the import statement, variables are just intended to make both test parameters and assert/event arguments configuration more flexible. However, they allow some basic arithmetic operations (+, -, *, /) with these restrictions:
• (-, *, /) allow numeric operands only.
• String + <value> treats <value> as a string value and return the string concatenation.
• In mixed expressions (like (1 + 2) + "hola"), numeric operations have precedence. So, (1 + 2) + "hola" = "3hola".
If you need more complex stuff about variables (within tests scope, for example), you might consider using Python embedded scripts.

## Resource References

Any string literal can contain one or more resource references. A resource reference is a way to say that a string literal content must be filled with a file content at the point where the resource reference is specified. The syntax is:
" ... ${path/to/file_1} ...${path/to/file_2} ... "
For instance,

• If the file.txt contains the text file content and if you define:
$var = "${./file.txt}";
Fressia will resolve the variable $var as: $var = "file content";
• If the file1.txt contains the text A and file2.txt contains the text B, if you define:
$var = "f1 + f1 is${./file1.txt} + ${./file2.txt}"; Fressia will resolve the variable $var as:
$var = "f1 + f2 is A + B"; Resource references are resolved just before a test is going to be executed. This is useful, for instance, when you want to communicate two tests. For example, in the suite below: • Tests send_hola and send_hola2 write the text hola and hola2 respectively to file.txt. • Both read and read_again tests check that file.txt contains the text hola.  suite pass_hola { test send_hola { [action] : embedded script; execute python (" file = open('./file.txt','w') file.write('hola') file.close() "); } test read { [action] : command; exec : "echo${file.txt}";
} asserts {
text contains ("hola");
}

test send_hola2 {
[action] : embedded script;
execute python ("

file = open('./file.txt','w')
file.write('hola2')
file.close()

");
}

// this test should fail cause
// the file content changed.

[action] : command;
exec : "echo hola";
} asserts {
text contains ("${file.txt}"); } }  This example may seem a little clumsy but please have in mind that test types can vary in a wide range if you consider that you can call external applications living in different environments. ## Actions The test action is indicated in the action block header. It starts with the label "[action]:" and it's followed by some reserved words as indicated below: ### HTTP(s)/REST Services These actions make a pure GET HTTP(s) call to the specified url: [action]: http call; [action]: https call; Required parameters: • url: the url that will be called. Allowed asserts: html, text. They are useful to check if a HTTP(s) service is up and responding. Example: suite just_to_test_google {$var = "http://www.google.com";

test test_html_integrity {
[action] : http call;
url : $var; } asserts { html isValid; } test test_page_title { [action] : http call; url :$var;
} asserts {
}

test test_both {
[action] : http call;
url : $var; } asserts { html isValid; text contains ("<title>Google</title>"); } } In case you are calling to a REST service you can use: [action]: rest call; Required parameters: • url: the url that will be called. Allowed asserts: xml. When you use this action you may want to check the response XML as shown in the example: suite rest_service_suite { test test_service { [action] : rest call; url : "http://your.service.address"; } asserts { xml isValid; xml validates ("/your/xsd/file"); } } If you need to make both an HTTP(s) or a REST client authenticated call you can use: [action]: client auth https call; Required parameters: • url: the url that will be called. • cert path: path to a PKC12 certificate. • cert pass: certificate password. Allowed asserts: xml, html, text. If you want to download a file from an URL you can do - with the same parameter as actions above: [action]: http download; [action]: https download; [action]: client auth https download; This case needs an extra parameter: • dest dir: an existing directory path where the file will be downloaded. ### Command-line Programs You can execute command line programs with: [action]: command; Required parameters: • exec: The command line that you want to execute. Allowed asserts: xml, text. Example: suite simple_command_tests { test check_file { [action] : command; exec : "ls /some/file.txt"; } asserts { not (text contains ("No such file or directory")); } test check_bash { [action] : command; exec : "sh /some/dir/print_ok.sh"; } asserts { text contains ("ok"); } } In case you want to execute a command that requires some user input you can use: [action]: blocking command; Required parameters: • exec: The command line that you want to execute. • user input: This is a string with the input values that you want to be injected when the program request for user interaction. They need to be as many as the number of input values the program requires and are injected in the same order you specify them. They can be all included in a single string separated by ; or you can add one clause per value. For instance, if you want to enter the values a and b you can write: user input : "a;b"; or user input : "a"; user input : "b";  Allowed asserts: xml, text. Example: Suppose you execute something from the command line like this: $ install-something.sh
are you sure? (yes|no):

At this point the program execution gets blocked waiting for the user response. The user types yes.
$install-something.sh are you sure? (yes|no): yes are you really sure? (yes|no):  And it gets blocked again. The user types yes again and the program finishes. $ install-something.sh
are you sure? (yes|no): yes
are you really sure? (yes|no): yes
ok, done.
$ If you want to test that if you reply yes to both questions you receive the message 'ok, done.', the suite would look like: suite blocking_command_tests { test check_flow { [action] : blocking command; exec : "install-something.sh"; user input : "yes"; user input : "yes"; } asserts { text contains ("ok, done."); } } ### Python Scripts Fressia is fully integrated with Jython 2.2.1. This means that you can execute Python 2.2.1 scripts, by directly embedding them into Fressia scripts or by using resource references, without needing to have a python installation somewhere. The action is [action]: embedded script; It doesn't require parameters. You specify the script you want to execute with the following event: execute python ("script code|resource reference") Examples: • if you execute the code below you would just get 'hola' printed out (and captured by Fressia). suite python_sample { test simple_print { [action] : embedded script; execute python ("print 'hola'"); } }  Note that if you use quotes for string literals, and your code is inserted into the Fressia test, you have to escape them with '\'. The previous example would look like: suite python_sample { test simple_print { [action] : embedded script; execute python ("print \"hola\""); } }  • You can also create a file with the code inside and call it with a resource reference. For example, if you have a python script file called hola.py somewhere (with the print 'hola' line inside) you can define: suite python_sample { test simple_print { [action] : embedded script; execute python ("${./path/to/hola.py}");
}

}

In this case you don't need to modify the code to escape the quotes.

#### Interacting with Fressia

In this context a python script becomes interesting if it can interacts with Fressia so that Fressia can be able to detect the script (test) status - for example, in the code above, how does Fressia knows if running that print is good or bad?

There are three ways to make your embedded python script interact with Fressia:
• Using asserts - you can use both xml and/or text asserts.

Example:
suite python_sample {

test simple_print {
[action] : embedded script;
execute python ("print 'hola'");
} asserts {
text contains ("hola");
}

}

• Setting up a 'tell_fressia' variable - if your script have a more complex flow maybe asserts will become too hard to handle. In this case you can use a variable, called 'tell_fressia' (which is a list), at any point of your code and set it up with this format:
tell_fressia = ['passed|failed', 'some message']
Fressia will consider the last assignment of this variable as your test status and will report the message you provided.

Example:
suite python_suite {

test fuzzy_test {
[action] : embedded script;
execute python
("

import random
a = random.randint(0,10)

if a >= 5:
tell_fressia = ['passed', 'cool']
else:
tell_fressia = ['failed', 'argh']

");

}

}

• Using a PyUnit script - this is the ideal way cause, on the one hand, PyUnit provides a nice tool set to deal with a wide range of testing scenarios and, on the other, Fressia can detect tests status and generate pretty good reports with the results.

Example:
suite python_suite {

test pyunit {
[action] : embedded script;
execute python
("

import unittest

class DummyIntsTestCase(unittest.TestCase):
self.assertEquals((1 + 2), 4)
self.assertEquals(0 + 1, 1)
def testMultiply(self):
self.assertEquals((0 * 10), 0)
self.assertEquals((5 * 8), 40)
if __name__ == '__main__':
unittest.main()

");

}

}

Note that when you use either the 'tell_fressia' variable or a PyUnit script you can still use Fressia asserts.

### Web Graphical User Interfaces

Fressia is integrated with both Selenium Server and Selenium Remote Control (RC). When you execute Fressia, a Selenium Server instance is brought up (configuration details here). You might use any Selenium RC driver to send events to that server instance but, since Fressia is integrated with Jython, the easiest way is to use the Selenium RC Python driver . Fressia provides the action below to make the Python driver code handling easier.
[action]: webgui events;
Fressia will use the following parameters to adjust your Python driver code:
• url: The page URL where the events will be triggered.
• browser: Browser that Fressia (Selenium) will open up to load the page indicated in 'url'. The string passed here must be a Selenium 'browser command'. At the moment, supported browser commands are:

• "*firefox" or "*chrome" for Firefox 2 and 3 in Chrome mode. This is the recommended mode when you are using Firefox.
• "*firefoxproxy" for Firefox 2 and 3 in normal mode.
• "*iexplore" or "*iehta" for Internet Explorer in HTA mode. This is the recommended mode when you are using Internet Explorer.
• "*iexploreproxy" for Internet Explorer in normal mode.
• "*safari" for Safari 2 and 3.
• "*opera" for Opera 8 and 9.
• "*konqueror" for Konqueror.

If your browser is installed in a non-standard path then you can use this format to indicate the path to it:

   "*<browser command> /path/to/your/browser-binary-executable"


If you want to trigger your events using a different Selenium Server (other than the server that is automatically brought up by Fressia) you can use these two optional parameters:

• selenium host (optional): Selenium server host name;
• selenium port (optional): Selenium server port
The Python driver code is specified in the same way as you do it for embedded python code (following the same rules to interact with Fressia) with:
execute python ("script code|resource reference")

Allowed asserts: xml, text.

Example:

The suite below will:
1. bring up a Firefox instance.
3. type 'hola' in the search box.
4. click on 'search' button.
5. wait 30 seconds for the results page to be loaded.
suite simple_google_webgui_events {

test search_hola {
[action] : webgui events;
browser: "*firefox";
execute python
("

from selenium import selenium

tell_fressia = ['failed', 'in case it does not finish.']

sel = selenium('', 0, '', '') # <-- you don't need to
sel.start()                   #     specify the selenium
#     server params here.

sel.open('/')
sel.type('q', 'hola')         # type 'hola' in the search
# box.

sel.click('btnG')             # click on 'search' button.

sel.stop()

tell_fressia = ['passed', 'well, at least it finished.']

");
}

}

Note that you don't need to specify the Selenium Server parameters in the code cause Fressia will adjust them according to what you have specified in the action parameters above. This is useful, for example, when you have a suite with global variables imported from many suites - action parameters might be assigned with global variables.

#### Fressia and Selenium IDE

Last example code looks quite simple. You probably noticed that the trickiest part is to locate the page elements - for example, how do you know that you can locate the Google page search text box element with the string 'q'?. Selenium RC drivers provide many ways to locate page elements - you can take a look at them here - and there are many Selenium tutorials out there which you can use to speed up learning how to master tests creation with Selenium RC drivers.

Fortunately, Selenium also provides a really sweet graphical tool, that is very easy to use, which allows you to create web GUI tests so fast. It's a Firefox plugin called Selenium IDE. There, you can quickly record your moves around a web page and save them as test cases (and suites) in an internal HTML format. Once you have a recorded test case you can open it and play it back with the same tool.

What does happen when you have too much recorded test cases? You can't do it by hand with Selenium IDE - cause it would mean to do it one by one. Well, there are many ways to automate the test cases execution - and there are also good tutorials about it.

But. A nice Selenium IDE feature is that you can export your recorded test cases to several languages - Java, PHP, Perl. And guess what? You can export your test cases to Python. Selenium IDE creates really nice PyUnit web GUI tests. And yes, you can call that code directly from Fressia using the webgui events action. So, there is a way to create Python web GUI tests for Fressia without needing to write even a single Python code line.

For example:
• If you follow this short tutorial you'll have both installed Selenium IDE and created your first test case.
• Once you have created your first test case you can go to File > Export Test Case As > Python Selenium RC and create a Python file. If you name your file as test_case.py you can:
• call it from Fressia with a test like:
suite selenium_ide_created {

test test_case {
[action] : webgui events;
url : "the url you used in Selenium IDE";
browser: "*firefox";        # or whatever
execute python ("${/path/to/test_case.py}"); } }  Note that: • you *don't* need to make any single change to your generated code cause Fressia will make the necessary adjustments automatically. • you are no longer forced to use Firefox to execute your tests - remember that if you use Selenium IDE to play back your test cases you are forced to use Firefox (it's an add-on). Finally, it's worth mentioning that with this action you can have your web GUI tests integrated to another non-web GUI tests within a single test suite, which is so useful when you are doing system testing, for example, in a rich application. For instance, your suite might look like: suite hybrid_suite { // run-registration-user-input test case test check_registration_page { [action] : webgui events; url : "your registration page url"; browser: "*firefox"; # or whatever execute python ("${.../some_test_case.py}");
}

// check the database status after registration
// process

test check_database {
[action] : some action;// a python script or
// an external command
// call, for example.
blah ...
}

}


### Email Reception

Fressia has an internal SMTP server that can be used to test email senders. The only thing you need to do is to configure your sender to use it - Fressia uses port 2526 by default to start up its SMTP server but that can be configured here. Combined with the email assert set you can test a pretty good number of situations.

The action syntax is:
[action]: email reception;
Allowed asserts: email.

Example:

Suppose that you have a dummy email sender, called send_mail.py, written in Pyhton. It would look like:
import smtplib
def mail(srvURL=None, srvPort=None, sender='', to='', subject='', text=''):

headers = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (sender, to, subject)
mailServer = smtplib.SMTP(srvURL, srvPort)
mailServer.sendmail(sender, to, message)
mailServer.quit()

mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')

If you call this code (with > python send_mail.py, for example) it would send an email by connecting to the server smtp.com at the standard port 25.

Now, to test this code with Fressia you would need to:
1. adjust send_mail.py code to use the Fressia SMTP server. In this case you just need to chage the line
mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')

to
mail('localhost', 2526, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')

2. make Fressia run send_mail.py in some way. For this example, you can use either a command action or an embedded Python action. The embedded Python action test would look like:
test python_email {
[action] : embedded script;
execute python ("${send_mail.py}"); }  3. use the email reception action (and its assert set) to check whether the Fressia SMTP server received the email or not. Finally, the suite would look like: suite email_suite { test exec_email_sender { [action] : embedded script; execute python ("${send_mail.py}");
}

test check_email_reception {
[action] : email reception;
} asserts {
messages count (1);
messages anyBodyContains("Message");
messages anySubjectContains("Test");
messages anySenderContains("from@foo.bar");
messages anyRecipientContains("to@foo.bar");
}

}


### Time Constrained Actions

Some times, when you are doing performance testing for example, you need that your test can be reported as failed if it doesn't satisfy some time constraint. For that case, every action accepts this optional parameter:

 expected time: <time in milliseconds (a float number)>; 

Example: the test below will be reported as failed.
suite tc {

test late_test {
[action] : command;
exec : "sleep 1";
expected time: 500.0;
}

}


• Line comments with //.
• Block comments enclosed by /* and */.

## Importing Suites

You can import a whole suite into another by using the following sentence:
import suite "</path/to/suite/file>";

It can be used anywhere within a suite scope and there are some points that you have to have in mind:
• If the imported suite contains a variable name that matches a variable name of the importing suite, the latter value will be replaced with the earlier value. For instance, suite B passes cause $var value is replaced by the value contained in the suite A. suite A {$var = 10;
}

suite B {
$var = 5; import suite "./suite_A.fs"; test value { [action] : command; exec : "echo " +$var;
} asserts {
text contains (10);
not text contains (5);
}
}

Note that suite A doesn't contain any test. So, with this feature you can have suite files that are just for global configuration handling.
• If the imported suite contains tests, the imported suite name will be added as a prefix to the imported test names with the separator '->'. For example,
suite A {
import suite "./suite_B.fs"

test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}
}

suite B {
import suite "./suite_C.fs"

test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}
}

suite C {

test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}

}

If you execute suite B Fressia will run and report it as containing two tests:
• C->say_hi
• say_hi

On the other hand, if you execute suite A Fressia will run and report it as containing three tests:
• B->C->say_hi
• B->say_hi
• say_hi

Note that the example above shows that you can have tests with the same name in different suites.
• With this feature, you can create quite complex suite execution trees. The only restriction is that you *can't* have cyclic imports. Consider this situation:
suite A {
import suite "./suite_B.fs";
}

suite B {
import suite "./suite_C.fs";
}

suite C {
import suite "./suite_A.fs";
}

If you execute suite A Fressia will complain saying that you have a cycle in the execution tree - observe that the example assumes that you have stored the suite A in a file called ./suite_A.fs.

On the other hand, if you are executing suite A but suite C looks like:
suite C {
import suite "./suite_D.fs";
}

and the file ./suite_D.fs contains a suite that looks like:
suite A {
...
}

Fressia will also complain saying a cycle in the execution tree was found. This means that you have to be careful with having repeated suite names in the execution tree.

## Executing Suite Scripts

At the moment Fressia is a command line program. So, in every case, you have to open up a terminal or console to execute a Fressia script.

The instructions below assume that you:

### Unix/Linux/Mac OS X

$<your install dir>/bin/fressia </path/to/suite1.fs> </path/to/suite2.fs> ...  or $ fressia </path/to/suite1.fs> </path/to/suite2.fs> ...

If you added <your install dir>/bin to the executable path.

Example:

If you open a console and do
$cd <your install dir>  you can run all the samples distributed with Fressia with: $ ./bin/fressia samples/all.fs

Note: Some tests require Firefox or Safari depending on your system.

### Windows

The windows package is distributed with a nice free console application. To start it up you just have to click on the Fressia icon called 'Fressia Console' on your desktop or go to Start > Fressia Testing Framework > Fressia Console . The console is opened at <your install dir> by default - you can move from that directory if you want. Then, to execute a Fressia script do:
<your install dir>\bin\fressia  <\path\to\suite1.fs> <\path\to\suite2.fs> ...

Example:

If you open up the Fressia Console, assuming that you installed Fressia in C:\fressia, you can run all the samples distributed with Fressia with:
C:\fressia>.\bin\fressia samples\all.fs

Note: Some tests require IE.

### Fressia Report

Fressia generates a subdirectory, called report, in the directory where it's being executed which contains:
• An index.html file that is the report main page. You can browse this report with any browser with a kind of decent Javascript support. It's pretty intuitive so the best way to learn how to use it is to click around and see what happens.
• A fressia.log file.
Important: IE* has some problems to display the Fressia HTML report. So, using Firefox or Google Chrome is highly recommended.

### Fressia Configuration File

In the Fressia install home directory there is a subdirectory called res that contains a file named fressia_config.xml. If you edit it you will see this:
<configuration>

<section name="general">
<entry key="log4j.configuration" value="$fressia.home$/res/log4j.props"/>
</section>

<section name="smtp">
<entry key="start" value="true" />
<entry key="port" value="2526" />
</section>

<section name="selenium">
<entry key="start" value="true" />
<entry key="port" value="4444" />
</section>

</configuration>

• Both smtp and selenium sections are to define whether you want to start up (and what port you want to use) the Fressia internal SMTP server and the Fressia Internal Selenium Server respectively.
• If you want the fressia.log file to have your own format you can specify your Log4j properties file in the key log4j.configuration of the section general.