Documentation
Table of Contents
Introduction
Fressia it's basically composed of:
- A really simple scripting language - the Fressia
language which is a DSL.
- An interpreter to execute scripts written in Fressia language.
- A results report generator.
In the following sections you'll find all what you need to write and
execute Fressia scripts.
Suites
A suite script file is a normal text file that follows this syntax:
suite <suite_name> {
// test definitions
// or
// suite import definitions
// or
// variable definitions
}
The suite name is user defined and is not restricted to be unique,
but if you have nested suites - using the
import sentence - you might have
some troubles if you have repeated names among several suites.
The file name can be anything and it doesn't have to be related to the suite name
- although, despite is not a restriction,
we recommend using the extension '
fs'.
Tests
A test is composed of the action block and the asserts block. General
syntax is:
test <test_name> {
[action]: <the test action definition>;
// parameters
// or
// events
} asserts {
// asserts
}
The test name is also user defined and it must be unique within the suite
it belongs to. The action definition just accepts the reserved words that
are explained in the sections below.
Both parameter and
event set are related to every specific action - some actions don't
need them - and follow the same structure.
Asserts and Events
Asserts are applied against
the action response after it has been executed while Events are
executed during the action execution. They:
- Implicitly return a
boolean value.
- Are all joined, as explained below, and
treated as a single implicit boolean expression. If this final
expression is
false
then the whole test is reported as failed.
- Are not strictly
related to actions but every action has an assert (event) set that is allowed
to accept.
The syntax of a single assert (event) expression is quite simple. It's specified
by two reserved words in this way:
In case of Asserts:
target_type condition ("argument");
- The target type is to indicate the 'data type' that Fressia will use
to handle the action response after the action execution.
- The condition is what you actually want to check in the response using
the information provided in the argument. The latter is not needed in some
cases; so some expressions just look like:
target_type condition;
In case of Events:
event_type event ("parameters");
- The first word is to indicate the 'event type' that Fressia will use
to handle the action event execution.
- The second word is the event that is going to be triggered during the action
execution using the information provided in the parameters. The latter is not needed in some
cases; so some expressions just look like:
event_type event;
They can be combined using these boolean operators:
-
and
(you can also use &&
).
-
or
(you can also use ||).
-
not
(you can also use !).
Examples:
not (t c ("a"));
!t1 c1 ("a1") && (t2 c2 ("a2") or t3 c3 ("a3"));
Single assert (event) expressions separated by semicolon are joined with the '&&'
operator. Hence, these two expressions are equivalent:
t c ("a");
e c ("b");
t c ("a") and e c ("b");
Text Asserts
Here, the action response is treated as flat text. They can be:
text contains ("some text");
text equals ("some text");
text matches ("some regular expression");
Regular expressions must be in the
Java
format.
XML/HTML Asserts
In this case the assert response is treated as an XML
DOM. You can have two of these asserts:
xml isValid;
Above is used to check if the action response is a well-formed XML document.
xml validates ("/path/to/an/XSD/file");
Above is used to check if the action response XML document validates an
XSD. Notice that XSD is a rich XML-ish language and, combined with the first
XML assert, it allows you to test anything from an XML document.
Besides, you can check if an HTML document is well-formed and
W3C compliant with:
html isValid;
Email Asserts
These are valid within the context of an
email action only. Here you don't have
an action response but the state of the Fressia internal SMTP server once it
has received (or not) a set of emails. So, these asserts are things that you
may wonder about that email set. They can be to:
Count the number of messages received by the internal SMTP server:
messages count ("some integer number");
Check if all messages contain a particular text in their body texts:
messages eachBodyContains ("some text");
Check if any message contains a particular text in its body text:
messages eachSubjectContains ("some text");
Check if all messages contain a particular text in their subject texts:
messages anyBodyContains ("some text");
Check if any message contains a particular text in its subject text:
messages anySubjectContains ("some text");
Check if any message sender ('From' header field) contains an email address:
messages anySenderContains("an email address");
Check if all message senders ('From' header field) contain an email address:
messages eachSenderContains ("an email address");
Check if any recipient ('To' header field) contains an email address:
messages anyRecipientContains ("an email address");
Check if all recipients ('To' header field) contain an email address:
messages eachRecipientContains ("an email address");
See an example
here.
Variables
You can define variables within the scope of a suite - variable definitions
are *not* allowed within a test scope.
The syntax is:
$<name> = <value>;
The value type can be:
String
- enclosed by quotes. Example: $var = "hola";
Integer
Float
Boolean
- true
or false
At the moment, combined with the
import
statement,
variables are just intended to make both test parameters
and assert/event arguments configuration more flexible. However, they
allow some basic arithmetic operations (
+
,
-
,
*
,
/
)
with these restrictions:
- (
-
, *
, /
) allow numeric operands only.
String + <value>
treats <value>
as a string value and return the string concatenation.
- In mixed expressions (like
(1 + 2) + "hola"
), numeric
operations have precedence. So, (1 + 2) + "hola" = "3hola"
.
If you need more complex stuff about variables (within tests scope, for example),
you might consider using Python embedded
scripts.
Resource References
Any string literal can contain one or more resource references. A resource
reference is a way to say that a string literal content must be
filled with a file content at the point where the resource reference is
specified. The syntax is:
" ... ${path/to/file_1} ... ${path/to/file_2} ... "
For instance,
Resource references are resolved just before a test is going to be executed.
This is useful, for instance, when you want to communicate two tests.
For example, in the suite below:
- Tests
send_hola
and send_hola2
write the text hola
and hola2
respectively
to file.txt
.
- Both
read
and read_again
tests check
that file.txt
contains the text hola
.
suite pass_hola {
test send_hola {
[action] : embedded script;
execute python ("
file = open('./file.txt','w')
file.write('hola')
file.close()
");
}
test read {
[action] : command;
exec : "echo ${file.txt}";
} asserts {
text contains ("hola");
}
test send_hola2 {
[action] : embedded script;
execute python ("
file = open('./file.txt','w')
file.write('hola2')
file.close()
");
}
// this test should fail cause
// the file content changed.
test read_again {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("${file.txt}");
}
}
This example may seem a little clumsy but please have in mind that test types can
vary in a wide range if you consider that you can call external
applications living in different environments.
Actions
The test action is indicated in the action block header. It starts with
the label "[action]:" and it's followed by some reserved words
as indicated below:
HTTP(s)/REST Services
These actions make a pure GET HTTP(s) call to the specified url:
[action]: http call;
[action]: https call;
Required parameters:
- url: the url that will be called.
Allowed asserts:
html,
text.
They are useful
to check if a HTTP(s) service is up and responding.
Example:
suite just_to_test_google {
$var = "http://www.google.com";
test test_html_integrity {
[action] : http call;
url : $var;
} asserts {
html isValid;
}
test test_page_title {
[action] : http call;
url : $var;
} asserts {
text contains ("<title>Google</title>");
}
test test_both {
[action] : http call;
url : $var;
} asserts {
html isValid;
text contains ("<title>Google</title>");
}
}
In case you are calling to a REST service you can use:
[action]: rest call;
Required parameters:
- url: the url that will be called.
Allowed asserts:
xml.
When you use this action you may want to check the response XML as shown in the example:
suite rest_service_suite {
test test_service {
[action] : rest call;
url : "http://your.service.address";
} asserts {
xml isValid;
xml validates ("/your/xsd/file");
}
}
If you need to make both an HTTP(s) or a REST client authenticated call you can use:
[action]: client auth https call;
Required parameters:
- url: the url that will be called.
- cert path: path to a PKC12 certificate.
- cert pass: certificate password.
Allowed asserts:
xml,
html,
text.
If you want to download a file from an URL you can do - with the same parameter as actions above:
[action]: http download;
[action]: https download;
[action]: client auth https download;
This case needs an extra parameter:
- dest dir: an existing directory path where the file will be downloaded.
Command-line Programs
You can execute command line programs with:
[action]: command;
Required parameters:
- exec: The command line that you want to execute.
Allowed asserts:
xml,
text.
Example:
suite simple_command_tests {
test check_file {
[action] : command;
exec : "ls /some/file.txt";
} asserts {
not (text contains ("No such file or directory"));
}
test check_bash {
[action] : command;
exec : "sh /some/dir/print_ok.sh";
} asserts {
text contains ("ok");
}
}
In case you want to execute a command that requires some user input you can use:
[action]: blocking command;
Required parameters:
Allowed asserts:
xml,
text.
Example:
Suppose you execute something from the command line like this:
$ install-something.sh
are you sure? (yes|no):
At this point the program execution gets blocked waiting for the user response. The user types
yes
.
$ install-something.sh
are you sure? (yes|no): yes
are you really sure? (yes|no):
And it gets blocked again. The user types
yes
again and the program finishes.
$ install-something.sh
are you sure? (yes|no): yes
are you really sure? (yes|no): yes
ok, done.
$
If you want to test that if you reply
yes
to both questions you receive the message '
ok, done.
',
the suite would look like:
suite blocking_command_tests {
test check_flow {
[action] : blocking command;
exec : "install-something.sh";
user input : "yes";
user input : "yes";
} asserts {
text contains ("ok, done.");
}
}
Python Scripts
Fressia is fully integrated with Jython 2.2.1. This means
that you can execute Python 2.2.1 scripts, by directly embedding
them into Fressia scripts or by using
resource references, without needing to
have a python installation somewhere. The action is
[action]: embedded script;
It doesn't require parameters. You specify the script you want to execute with
the following event:
execute python ("script code|resource reference")
Examples:
- if you execute the code below you would just get 'hola' printed out (and
captured by Fressia).
suite python_sample {
test simple_print {
[action] : embedded script;
execute python ("print 'hola'");
}
}
Note that if you use quotes for string literals, and your code is inserted into the Fressia test,
you have to escape them with '\'
.
The previous example would look like:
suite python_sample {
test simple_print {
[action] : embedded script;
execute python ("print \"hola\"");
}
}
- You can also create a file with the code inside and call it with a resource reference. For example,
if you have a python script file called
hola.py
somewhere (with the print 'hola'
line inside) you can
define:
suite python_sample {
test simple_print {
[action] : embedded script;
execute python ("${./path/to/hola.py}");
}
}
In this case you don't need to modify the code to escape the quotes.
Interacting with Fressia
In this context a python script becomes interesting if it can interacts
with Fressia so that Fressia can be able to detect the script (test) status - for example, in
the code above, how does Fressia knows if running that print is good or bad?
There are three ways to make your embedded python script interact with Fressia:
- Using asserts - you can use both xml and/or
text asserts.
Example:
suite python_sample {
test simple_print {
[action] : embedded script;
execute python ("print 'hola'");
} asserts {
text contains ("hola");
}
}
- Setting up a 'tell_fressia' variable - if your script have a more complex flow maybe
asserts will become too hard to handle. In this case you can use a variable, called 'tell_fressia'
(which is a list), at any point of your code and set it up with this format:
tell_fressia = ['passed|failed', 'some message']
Fressia will consider the last assignment of this variable as your test status and will
report the message you provided.
Example:
suite python_suite {
test fuzzy_test {
[action] : embedded script;
execute python
("
import random
a = random.randint(0,10)
if a >= 5:
tell_fressia = ['passed', 'cool']
else:
tell_fressia = ['failed', 'argh']
");
}
}
- Using a PyUnit script - this is the ideal way cause, on the one hand, PyUnit provides
a nice tool set to deal with a wide range of testing scenarios and, on the other,
Fressia can detect tests status and generate pretty good reports with the results.
Example:
suite python_suite {
test pyunit {
[action] : embedded script;
execute python
("
import unittest
class DummyIntsTestCase(unittest.TestCase):
def testAdd(self):
self.assertEquals((1 + 2), 4)
self.assertEquals(0 + 1, 1)
def testMultiply(self):
self.assertEquals((0 * 10), 0)
self.assertEquals((5 * 8), 40)
if __name__ == '__main__':
unittest.main()
");
}
}
Note that when you use either the 'tell_fressia' variable or a PyUnit script you can still use
Fressia asserts.
Web Graphical User Interfaces
Fressia is integrated with both
Selenium
Server and
Selenium Remote Control (RC).
When you execute Fressia, a Selenium Server
instance is brought up (configuration details
here).
You might use any Selenium RC driver to send
events to that server instance but, since Fressia is integrated with
Jython,
the easiest way is to use the Selenium RC
Python driver
. Fressia provides the action below to make the Python driver code handling easier.
[action]: webgui events;
Fressia will use the following parameters to adjust your Python driver code:
The Python driver code is specified in the same way as you do it for
embedded python code (following the same rules to interact with Fressia) with:
execute python ("script code|resource reference")
Allowed asserts:
xml,
text.
Example:
The suite below will:
- bring up a Firefox instance.
- open up Google.
- type 'hola' in the search box.
- click on 'search' button.
- wait 30 seconds for the results page to be loaded.
suite simple_google_webgui_events {
test search_hola {
[action] : webgui events;
url : "http://www.google.com";
browser: "*firefox";
execute python
("
from selenium import selenium
tell_fressia = ['failed', 'in case it does not finish.']
sel = selenium('', 0, '', '') # <-- you don't need to
sel.start() # specify the selenium
# server params here.
sel.open('/')
sel.type('q', 'hola') # type 'hola' in the search
# box.
sel.click('btnG') # click on 'search' button.
sel.wait_for_page_to_load(30000)
sel.stop()
tell_fressia = ['passed', 'well, at least it finished.']
");
}
}
Note that you don't need to specify the Selenium Server parameters in the code cause
Fressia will adjust them according to what you have specified in the action parameters above. This
is useful, for example, when you have a suite with global variables imported from many suites - action
parameters might be assigned with global variables.
Fressia and Selenium IDE
Last example code looks quite simple. You probably noticed that the trickiest part is to
locate the page elements - for example, how do you know that you can
locate the Google page search text box element with the string 'q'?. Selenium RC drivers provide
many ways to locate page elements - you can take a look at them
here - and
there are many Selenium tutorials out there which you can use to speed up learning how to master
tests creation with Selenium RC drivers.
Fortunately, Selenium also provides a really sweet graphical
tool, that is very easy to use, which allows you to create web GUI tests so fast. It's a Firefox
plugin called Selenium IDE. There, you can quickly record your moves around a web page and save
them as test cases (and suites) in an internal HTML format. Once you have a recorded test case you can
open it and play it back with the same tool.
What does happen when you have too
much recorded test cases? You can't do it by hand with Selenium IDE - cause it would mean to do it
one by one. Well, there are many
ways to automate the test cases execution - and there are also good tutorials about it.
But. A nice Selenium IDE feature is that you can export your recorded test cases to several languages -
Java, PHP, Perl. And guess what? You can export your test cases to Python. Selenium IDE creates
really nice PyUnit web GUI tests. And yes, you can call that code directly from Fressia using the
webgui events
action. So, there is a way to create Python web GUI tests for Fressia without
needing to write even a single Python code line.
For example:
Finally, it's worth mentioning that with this action you can have your web GUI tests integrated to another non-web GUI tests within
a single test suite, which is so
useful when you are doing system testing, for example, in a rich application. For instance, your
suite might look like:
suite hybrid_suite {
// run-registration-user-input test case
test check_registration_page {
[action] : webgui events;
url : "your registration page url";
browser: "*firefox"; # or whatever
execute python ("${.../some_test_case.py}");
}
// check the database status after registration
// process
test check_database {
[action] : some action;// a python script or
// an external command
// call, for example.
blah ...
}
}
Email Reception
Fressia has an internal SMTP server that can be used
to test email senders. The only thing you need to do is
to configure your sender to use it - Fressia uses port 2526 by default
to start up its SMTP server but that can be configured
here. Combined with the
email assert
set you can test a pretty good number of situations.
The action syntax is:
[action]: email reception;
Allowed asserts:
email.
Example:
Suppose that you have a dummy email sender, called
send_mail.py
,
written in Pyhton. It would look like:
import smtplib
def mail(srvURL=None, srvPort=None, sender='', to='', subject='', text=''):
headers = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (sender, to, subject)
message = headers + text
mailServer = smtplib.SMTP(srvURL, srvPort)
mailServer.sendmail(sender, to, message)
mailServer.quit()
mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
If you call this code (with
> python send_mail.py
, for example) it would
send an email by connecting to the server
smtp.com
at the
standard port 25.
Now, to test this code with Fressia you would need to:
- adjust
send_mail.py
code to use the Fressia SMTP server.
In this case you just need to chage the line
mail('smtp.com', 25, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
to
mail('localhost', 2526, 'from@foo.bar', 'to@foo.bar', 'Test', 'Message')
- make Fressia run
send_mail.py
in some way. For this example,
you can use either a command action or an
embedded Python action. The embedded Python
action test would look like:
test python_email {
[action] : embedded script;
execute python ("${send_mail.py}");
}
- use the
email reception
action (and its
assert set) to check
whether the Fressia SMTP server received the email or not.
Finally, the suite would look like:
suite email_suite {
test exec_email_sender {
[action] : embedded script;
execute python ("${send_mail.py}");
}
test check_email_reception {
[action] : email reception;
} asserts {
messages count (1);
messages anyBodyContains("Message");
messages anySubjectContains("Test");
messages anySenderContains("from@foo.bar");
messages anyRecipientContains("to@foo.bar");
}
}
Time Constrained Actions
Some times, when you are doing performance testing for example, you
need that your test can be reported as failed if it doesn't satisfy some
time constraint. For that case, every action accepts this optional parameter:
expected time: <time in milliseconds (a float number)>;
Example: the test below will be reported as failed.
suite tc {
test late_test {
[action] : command;
exec : "sleep 1";
expected time: 500.0;
}
}
Suites Comments
As you probably noticed in the examples above, you can add comments at
any point in a Fressia suite. They can be:
-
Line comments with
//
.
-
Block comments enclosed by
/*
and */
.
Importing Suites
You can import a whole suite into another by using the following
sentence:
import suite "</path/to/suite/file>";
It can be used anywhere within a suite scope and there are some points
that you have to have in mind:
- If the imported suite contains a variable name that matches a
variable name of the importing suite, the latter value will be replaced
with the earlier value. For instance, suite
B
passes cause
$var
value is replaced by the value contained in the suite
A
.
suite A {
$var = 10;
}
suite B {
$var = 5;
import suite "./suite_A.fs";
test value {
[action] : command;
exec : "echo " + $var;
} asserts {
text contains (10);
not text contains (5);
}
}
Note that suite A
doesn't contain any test. So, with this feature
you can have suite files that are just for global configuration
handling.
- If the imported suite contains tests, the imported suite name will
be added as a prefix to the imported test names with the separator
'->'. For example,
suite A {
import suite "./suite_B.fs"
test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}
}
suite B {
import suite "./suite_C.fs"
test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}
}
suite C {
test say_hi {
[action] : command;
exec : "echo hola";
} asserts {
text contains ("hola");
}
}
If you execute suite B
Fressia will run and report it as
containing two tests:
On the other hand,
if you execute suite A
Fressia will run and report it as containing three tests:
B->C->say_hi
B->say_hi
say_hi
Note that the example above shows that you can have tests with the same
name in different suites.
-
With this feature, you can create quite complex suite execution trees.
The only restriction is that you *can't* have cyclic imports. Consider
this situation:
suite A {
import suite "./suite_B.fs";
}
suite B {
import suite "./suite_C.fs";
}
suite C {
import suite "./suite_A.fs";
}
If you execute suite A
Fressia will complain saying that
you have a cycle in the execution tree - observe that the example assumes
that you have stored the suite A
in a file called
./suite_A.fs
.
On the other hand, if you are executing suite A
but suite C
looks like:
suite C {
import suite "./suite_D.fs";
}
and the file ./suite_D.fs
contains a suite that looks like:
suite A {
...
}
Fressia will also complain saying a cycle in the execution tree was found. This
means that you have to be careful with having repeated suite names in the
execution tree.
Executing Suite Scripts
At the moment Fressia is a command line program. So, in every case, you have
to open up a terminal or console to execute a Fressia script.
The instructions below assume that you:
Unix/Linux/Mac OS X
$<your install dir>
/bin/fressia </path/to/suite1.fs> </path/to/suite2.fs> ...
or
$ fressia </path/to/suite1.fs> </path/to/suite2.fs> ...
If you added
<your install dir>/bin
to the executable path.
Example:
If you open a console and do
$ cd <your install dir>
you can run all the samples distributed with Fressia with:
$ ./bin/fressia samples/all.fs
Note: Some tests require Firefox or Safari depending on your system.
Windows
The windows package is distributed with a nice
free console application.
To start it up you just have to click on the Fressia icon called 'Fressia Console' on your desktop
or go to
Start > Fressia Testing Framework > Fressia Console
. The console
is opened at
<your install dir>
by default - you can move from that
directory if you want. Then, to execute a Fressia script do:
<your install dir>
\bin\fressia <\path\to\suite1.fs> <\path\to\suite2.fs> ...
Example:
If you open up the Fressia Console, assuming that you installed Fressia
in
C:\fressia
, you can run all the samples distributed with Fressia with:
C:\fressia>.\bin\fressia samples\all.fs
Note: Some tests require IE.
Fressia Report
Fressia generates a subdirectory, called
report
, in the
directory where it's being executed which contains:
- An
index.html
file that is the report main page. You
can browse this report with any browser with a kind of decent Javascript support. It's
pretty intuitive so the best way to learn how to use it is to click around and see
what happens.
- A
fressia.log
file.
Important: IE* has some problems to display the Fressia HTML report. So, using
Firefox or Google Chrome is highly recommended.
Fressia Configuration File
In the Fressia install home directory there is a subdirectory called
res
that contains a file named
fressia_config.xml
. If you
edit it you will see this:
<configuration>
<section name="general">
<entry key="log4j.configuration" value="$fressia.home$/res/log4j.props"/>
</section>
<section name="smtp">
<entry key="start" value="true" />
<entry key="port" value="2526" />
</section>
<section name="selenium">
<entry key="start" value="true" />
<entry key="port" value="4444" />
</section>
</configuration>
- Both
smtp
and selenium
sections are to define whether you
want to start up (and what port you want to use) the Fressia internal SMTP server and
the Fressia Internal Selenium Server respectively.
- If you want the
fressia.log
file to have your own format
you can specify your Log4j properties file in the key log4j.configuration
of the section general
.