Testing Framework

Documentation


Table of Contents

Introduction

Fressia it's basically composed of: In the following sections you'll find all what you need to write and execute Fressia scripts.

Suites

A suite script file is a normal text file that follows this syntax:
suite <suite_name> {
   
  // test definitions
  //       or
  // suite import definitions
  //       or
  // variable definitions
  
}
The suite name is user defined and is not restricted to be unique, but if you have nested suites - using the import sentence - you might have some troubles if you have repeated names among several suites.

The file name can be anything and it doesn't have to be related to the suite name - although, despite is not a restriction, we recommend using the extension 'fs'.

Tests

A test is composed of the action block and the asserts block. General syntax is:
test <test_name> {
    [action]: <the test action definition>;
    
    // parameters
    //   or
    // events
    
} asserts {
    
    // asserts
     
}
The test name is also user defined and it must be unique within the suite it belongs to. The action definition just accepts the reserved words that are explained in the sections below.

Both parameter and event set are related to every specific action - some actions don't need them - and follow the same structure.

Asserts and Events

Asserts are applied against the action response after it has been executed while Events are executed during the action execution. They: The syntax of a single assert (event) expression is quite simple. It's specified by two reserved words in this way:

In case of Asserts:
target_type condition ("argument");
In case of Events:
event_type event ("parameters");
They can be combined using these boolean operators: Examples:
not (t c ("a"));
!t1 c1 ("a1") && (t2 c2 ("a2") or t3 c3 ("a3"));
   
Single assert (event) expressions separated by semicolon are joined with the '&&' operator. Hence, these two expressions are equivalent:
t c ("a");
e c ("b");   
   
t c ("a") and e c ("b");   
   

Text Asserts

Here, the action response is treated as flat text. They can be:
text contains ("some text");
text equals ("some text");
text matches ("some regular expression");
Regular expressions must be in the Java format.

XML/HTML Asserts

In this case the assert response is treated as an XML DOM. You can have two of these asserts:
xml isValid;
Above is used to check if the action response is a well-formed XML document.
xml validates ("/path/to/an/XSD/file");
Above is used to check if the action response XML document validates an XSD. Notice that XSD is a rich XML-ish language and, combined with the first XML assert, it allows you to test anything from an XML document.

Besides, you can check if an HTML document is well-formed and W3C compliant with:
html isValid;

Email Asserts

These are valid within the context of an email action only. Here you don't have an action response but the state of the Fressia internal SMTP server once it has received (or not) a set of emails. So, these asserts are things that you may wonder about that email set. They can be to:

Count the number of messages received by the internal SMTP server:
messages count ("some integer number");

Check if all messages contain a particular text in their body texts:
messages eachBodyContains ("some text");

Check if any message contains a particular text in its body text:
messages eachSubjectContains ("some text");

Check if all messages contain a particular text in their subject texts:
messages anyBodyContains ("some text");

Check if any message contains a particular text in its subject text:
messages anySubjectContains ("some text");

Check if any message sender ('From' header field) contains an email address:
messages anySenderContains("an email address");

Check if all message senders ('From' header field) contain an email address:
messages eachSenderContains ("an email address");

Check if any recipient ('To' header field) contains an email address:
messages anyRecipientContains ("an email address");

Check if all recipients ('To' header field) contain an email address:
messages eachRecipientContains ("an email address");
See an example here.

Variables

You can define variables within the scope of a suite - variable definitions are *not* allowed within a test scope. The syntax is:
$<name> = <value>;
The value type can be: At the moment, combined with the import statement, variables are just intended to make both test parameters and assert/event arguments configuration more flexible. However, they allow some basic arithmetic operations (+, -, *, /) with these restrictions: If you need more complex stuff about variables (within tests scope, for example), you might consider using Python embedded scripts.

Resource References

Any string literal can contain one or more resource references. A resource reference is a way to say that a string literal content must be filled with a file content at the point where the resource reference is specified. The syntax is:
" ... ${path/to/file_1} ... ${path/to/file_2} ... "
For instance,

Resource references are resolved just before a test is going to be executed. This is useful, for instance, when you want to communicate two tests. For example, in the suite below:
 
suite pass_hola {
   
    test send_hola {
     [action] : embedded script;
     execute python ("

file = open('./file.txt','w')
file.write('hola')
file.close()
	 
	 ");
    } 

    test read {
      [action] : command;
      exec : "echo ${file.txt}";
    } asserts {
      text contains ("hola");	
    }
	
	
	
    test send_hola2 {
     [action] : embedded script;
     execute python ("

file = open('./file.txt','w')
file.write('hola2')
file.close()
	 
	 ");
    } 

    // this test should fail cause 
    // the file content changed.
	
    test read_again {
      [action] : command;
      exec : "echo hola";
    } asserts {
      text contains ("${file.txt}");	
    }	
	
}
 
This example may seem a little clumsy but please have in mind that test types can vary in a wide range if you consider that you can call external applications living in different environments.

Actions

The test action is indicated in the action block header. It starts with the label "[action]:" and it's followed by some reserved words as indicated below:

HTTP(s)/REST Services

These actions make a pure GET HTTP(s) call to the specified url:
[action]: http call;
[action]: https call;
Required parameters: Allowed asserts: html, text.

They are useful to check if a HTTP(s) service is up and responding.

Example:
suite just_to_test_google {

 $var = "http://www.google.com";
   
 test test_html_integrity {
  [action] : http call;
  url : $var;
 } asserts {
  html isValid;
 }

 test test_page_title {
  [action] : http call;
  url : $var;
 } asserts {
  text contains ("<title>Google</title>");
 }

 test test_both {
  [action] : http call;
  url : $var;
 } asserts {
  html isValid;
  text contains ("<title>Google</title>");
 }

}

In case you are calling to a REST service you can use:
[action]: rest call;
Required parameters: Allowed asserts: xml.

When you use this action you may want to check the response XML as shown in the example:
suite rest_service_suite {
   
 test test_service {
  [action] : rest call;
  url : "http://your.service.address";
 } asserts {
  xml isValid;
  xml validates ("/your/xsd/file");
 }

}
If you need to make both an HTTP(s) or a REST client authenticated call you can use:
[action]: client auth https call;
Required parameters: Allowed asserts: xml, html, text.

If you want to download a file from an URL you can do - with the same parameter as actions above:
[action]: http download;
[action]: https download;
[action]: client auth https download;
This case needs an extra parameter:

Command-line Programs

You can execute command line programs with:
[action]: command;
Required parameters: Allowed asserts: xml, text.

Example:
suite simple_command_tests {

 test check_file {
  [action] : command;
  exec : "ls /some/file.txt";
 } asserts {
  not (text contains ("No such file or directory"));
 }
 
 test check_bash {
  [action] : command;
  exec : "sh /some/dir/print_ok.sh";
 } asserts {
  text contains ("ok");
 } 

}
In case you want to execute a command that requires some user input you can use:
[action]: blocking command;
Required parameters: Allowed asserts: xml, text.

Example:

Suppose you execute something from the command line like this:
$ install-something.sh
are you sure? (yes|no): 
At this point the program execution gets blocked waiting for the user response. The user types yes.
$ install-something.sh
are you sure? (yes|no): yes
are you really sure? (yes|no): 
And it gets blocked again. The user types yes again and the program finishes.
$ install-something.sh
are you sure? (yes|no): yes
are you really sure? (yes|no): yes
ok, done.
$
If you want to test that if you reply yes to both questions you receive the message 'ok, done.', the suite would look like:
suite blocking_command_tests {

 test check_flow {
  [action] : blocking command;
  exec : "install-something.sh";
  user input : "yes";
  user input : "yes";
 } asserts {
  text contains ("ok, done.");
 }

}

Python Scripts

Fressia is fully integrated with Jython 2.2.1. This means that you can execute Python 2.2.1 scripts, by directly embedding them into Fressia scripts or by using resource references, without needing to have a python installation somewhere. The action is
[action]: embedded script;
It doesn't require parameters. You specify the script you want to execute with the following event:
execute python ("script code|resource reference")
Examples:

Interacting with Fressia

In this context a python script becomes interesting if it can interacts with Fressia so that Fressia can be able to detect the script (test) status - for example, in the code above, how does Fressia knows if running that print is good or bad?

There are three ways to make your embedded python script interact with Fressia: Note that when you use either the 'tell_fressia' variable or a PyUnit script you can still use Fressia asserts.

Web Graphical User Interfaces

Fressia is integrated with both Selenium Server and Selenium Remote Control (RC). When you execute Fressia, a Selenium Server instance is brought up (configuration details here). You might use any Selenium RC driver to send events to that server instance but, since Fressia is integrated with Jython, the easiest way is to use the Selenium RC Python driver . Fressia provides the action below to make the Python driver code handling easier.
[action]: webgui events;
Fressia will use the following parameters to adjust your Python driver code: The Python driver code is specified in the same way as you do it for embedded python code (following the same rules to interact with Fressia) with:
execute python ("script code|resource reference")


Allowed asserts: xml, text.

Example:

The suite below will:
  1. bring up a Firefox instance.
  2. open up Google.
  3. type 'hola' in the search box.
  4. click on 'search' button.
  5. wait 30 seconds for the results page to be loaded.
suite simple_google_webgui_events {

  test search_hola {
   [action] : webgui events;
    url : "http://www.google.com";
    browser: "*firefox";
    execute python
("

from selenium import selenium

tell_fressia = ['failed', 'in case it does not finish.']

sel = selenium('', 0, '', '') # <-- you don't need to
sel.start()                   #     specify the selenium  
                              #     server params here. 

sel.open('/')                 
sel.type('q', 'hola')         # type 'hola' in the search
                              # box.
                              
sel.click('btnG')             # click on 'search' button.

sel.wait_for_page_to_load(30000)
sel.stop()

tell_fressia = ['passed', 'well, at least it finished.']

");
   }

}
Note that you don't need to specify the Selenium Server parameters in the code cause Fressia will adjust them according to what you have specified in the action parameters above. This is useful, for example, when you have a suite with global variables imported from many suites - action parameters might be assigned with global variables.

Fressia and Selenium IDE

Last example code looks quite simple. You probably noticed that the trickiest part is to locate the page elements - for example, how do you know that you can locate the Google page search text box element with the string 'q'?. Selenium RC drivers provide many ways to locate page elements - you can take a look at them here - and there are many Selenium tutorials out there which you can use to speed up learning how to master tests creation with Selenium RC drivers.

Fortunately, Selenium also provides a really sweet graphical tool, that is very easy to use, which allows you to create web GUI tests so fast. It's a Firefox plugin called Selenium IDE. There, you can quickly record your moves around a web page and save them as test cases (and suites) in an internal HTML format. Once you have a recorded test case you can open it and play it back with the same tool.

What does happen when you have too much recorded test cases? You can't do it by hand with Selenium IDE - cause it would mean to do it one by one. Well, there are many ways to automate the test cases execution - and there are also good tutorials about it.

But. A nice Selenium IDE feature is that you can export your recorded test cases to several languages - Java, PHP, Perl. And guess what? You can export your test cases to Python. Selenium IDE creates really nice PyUnit web GUI tests. And yes, you can call that code directly from Fressia using the webgui events action. So, there is a way to create Python web GUI tests for Fressia without needing to write even a single Python code line.

For example:
Finally, it's worth mentioning that with this action you can have your web GUI tests integrated to another non-web GUI tests within a single test suite, which is so useful when you are doing system testing, for example, in a rich application. For instance, your suite might look like:
suite hybrid_suite {

 // run-registration-user-input test case

 test check_registration_page {
  [action] : webgui events;
   url : "your registration page url";
   browser: "*firefox";        # or whatever
   execute python ("${.../some_test_case.py}");
 }
   
 // check the database status after registration 
 // process
 
 test check_database {
  [action] : some action;// a python script or 
                         // an external command
                         // call, for example.
  blah ...
 }

}

Email Reception

Fressia has an internal SMTP server that can be used to test email senders. The only thing you need to do is to configure your sender to use it - Fressia uses port 2526 by default to start up its SMTP server but that can be configured here. Combined with the email assert set you can test a pretty good number of situations.

The action syntax is:
[action]: email reception;
Allowed asserts: email.

Example:

Suppose that you have a dummy email sender, called send_mail.py, written in Pyhton. It would look like:
import smtplib
def mail(srvURL=None, srvPort=None, sender='', to='', subject='', text=''):

  headers = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (sender, to, subject)
  message = headers + text
  mailServer = smtplib.SMTP(srvURL, srvPort)
  mailServer.sendmail(sender, to, message)
  mailServer.quit()

mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
If you call this code (with > python send_mail.py, for example) it would send an email by connecting to the server smtp.com at the standard port 25.

Now, to test this code with Fressia you would need to:
  1. adjust send_mail.py code to use the Fressia SMTP server. In this case you just need to chage the line
    mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
    
    to
    mail('localhost', 2526, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
    
  2. make Fressia run send_mail.py in some way. For this example, you can use either a command action or an embedded Python action. The embedded Python action test would look like:
    test python_email {
      [action] : embedded script;
      execute python ("${send_mail.py}");
    }
    
  3. use the email reception action (and its assert set) to check whether the Fressia SMTP server received the email or not.
Finally, the suite would look like:
suite email_suite {

 test exec_email_sender {
   [action] : embedded script;
   execute python ("${send_mail.py}");
 }


 test check_email_reception {
  [action] : email reception;
 } asserts {
  messages count (1);
  messages anyBodyContains("Message");
  messages anySubjectContains("Test");
  messages anySenderContains("from@foo.bar");
  messages anyRecipientContains("to@foo.bar");
 }


}

Time Constrained Actions

Some times, when you are doing performance testing for example, you need that your test can be reported as failed if it doesn't satisfy some time constraint. For that case, every action accepts this optional parameter:

expected time: <time in milliseconds (a float number)>;

Example: the test below will be reported as failed.
suite tc {

test late_test {
  [action] : command;
  exec : "sleep 1";
  expected time: 500.0;
 }
 
}

Suites Comments

As you probably noticed in the examples above, you can add comments at any point in a Fressia suite. They can be:

Importing Suites

You can import a whole suite into another by using the following sentence:
import suite "</path/to/suite/file>";
It can be used anywhere within a suite scope and there are some points that you have to have in mind:

Executing Suite Scripts

At the moment Fressia is a command line program. So, in every case, you have to open up a terminal or console to execute a Fressia script.

The instructions below assume that you:

Unix/Linux/Mac OS X

$<your install dir>/bin/fressia </path/to/suite1.fs> </path/to/suite2.fs> ...
or
$ fressia </path/to/suite1.fs> </path/to/suite2.fs> ...
If you added <your install dir>/bin to the executable path.

Example:

If you open a console and do
$ cd <your install dir>
you can run all the samples distributed with Fressia with:
$ ./bin/fressia samples/all.fs
Note: Some tests require Firefox or Safari depending on your system.

Windows

The windows package is distributed with a nice free console application. To start it up you just have to click on the Fressia icon called 'Fressia Console' on your desktop or go to Start > Fressia Testing Framework > Fressia Console . The console is opened at <your install dir> by default - you can move from that directory if you want. Then, to execute a Fressia script do:
<your install dir>\bin\fressia  <\path\to\suite1.fs> <\path\to\suite2.fs> ...
Example:

If you open up the Fressia Console, assuming that you installed Fressia in C:\fressia, you can run all the samples distributed with Fressia with:
C:\fressia>.\bin\fressia samples\all.fs
Note: Some tests require IE.

Fressia Report

Fressia generates a subdirectory, called report, in the directory where it's being executed which contains: Important: IE* has some problems to display the Fressia HTML report. So, using Firefox or Google Chrome is highly recommended.

Fressia Configuration File

In the Fressia install home directory there is a subdirectory called res that contains a file named fressia_config.xml. If you edit it you will see this:
<configuration>

  <section name="general">
     <entry key="log4j.configuration" value="$fressia.home$/res/log4j.props"/>
  </section>

  <section name="smtp">
     <entry key="start" value="true" />
     <entry key="port" value="2526" />
  </section>

  <section name="selenium">
     <entry key="start" value="true" />
     <entry key="port" value="4444" />
  </section>

</configuration>