Logix-Test is a Logix powered unit testing framework. It is available as a separate download from the Logix download page, and installs as the package ltest. The package consists of two major components. A command-line test-runner (ltest.runner), and a test language (ltest.testlang) which is an extension of Standard Logix.
The test language provides:
With test-driven-development the ratio of test code to application code often approaches one-to-one. With so much test code being written, there is a great deal to be gained by using a language tailored to the task. Tests developed using Logix-Test are considerably more compact, easier to maintain and more readable than tests written in a general purpose language.
For more examples of the features described in this section, see the unit-tests for Logix itself, which are written using the ltest framework. They are included in the standard distribution.
Across various unit-testing frameworks, the structure of tests and test-suites has a lot in common. Testing always consists of three stages
In practice, the test operations and assertions are often intermingled.
In some cases, a fourth stage is required to keep the state of the system clean for subsequent tests
The state created by the setup phase is sometimes called the fixture. Because a number of tests will commonly require a similar fixture, frameworks typically provide mechanisms to re-use the setup phase.
In the standard Python unit-test framework, for example, the fixture is created in a method of a class. The class may then contain multiple test methods, each of which share the same fixture. The fixture may also be inherited and extended in other test classes using regular Python inheritance.
The ltest framework provides similar facilities but is not class based. Instead it used nested scopes to make test definition much more concise.
The overall structure of an ltest file is:
limport ltest.testlang
setlang testlang.testlang
import <stuff-to-be-tested>
defsuite <suite-name>:
<setup>
deftest myFirstTest:
<test
operations>
<test
assertions>
deftest myOtherTest:
<test
operations>
<test
assertions>
<teardown>
For example, here are some tests on the built-in string type.
limport ltest.testlang
setlang testlang.testlang
defsuite main:
print "Setup"
empty = ""
one = "a"
a = "ho hum"
deftest append:
print "In test:
append"
empty + "!" ?= "!"
one + "!" ?=
"a!"
a +
"!" ?= "ho hum!"
deftest count:
print "In test:
count"
empty.count 'x' ?=
0
one.count 'a' ?= 1
one.count 'b' ?= 0
a.count 'x' ?= 0
a.count 'o' ?= 1
a.count 'h' ?= 2
print "Teardown"
print
There are a few things to note here.
The framework is designed to be use from the Logix prompt. The test file is simply a regular module. The suite can be run be calling it:
[std]: limport examples.stringtest
[std]: stringtest.main()
Setup
In test: append
Teardown
Setup
In test: count
Teardown
Ran 2 tests, 9 assertions, with 0 failures
From the print statements, we can see the tests were executed in the order they appear in the source-code, and that the setup and teardown code was executed before and after each test.
Another level of structure can be introduced with defgroup:
defsuite main:
print "Setup"
empty = ""
one = "a"
a = "ho hum"
deftest append:
print "In test:
append"
...
deftest count:
print "In test:
count"
...
defgroup unicode:
print "Setup
unicode"
uempty = unicode
empty
uone = unicode one
ua = unicode a
deftest backToStr:
print "In test: backToStr"
str uempty ?= empty
str uone ?= one
str ua ?= a
print "Teardown
unicode"
print "Teardown"
print
defgroup introduces a nested set of tests with their own setup and teardown.
For each test in the sub-group, both the top-level setup and the group setup will run, then the test runs, and finally the group teardown and top-level teardown runs.
[std]: limport examples.stringtest2
[std]: stringtest2.main()
Setup
In test: append
Teardown
Setup
In test: count
Teardown
Setup
Setup unicode
In test: backToStr
Teardown unicode
Teardown
Ran 3 tests, 12 assertions, with 0 failures
Test groups created with defgroup can also contain further sub-groups, nested to any depth.
As we have seen, the test-runner is used from the interactive prompt. It is started by invoking the required suite function.
If a test fails, the code around the problem is displayed, along with any message associated with the failure. The test-runner then presents a prompt. The prompt recognizes a number of simple commands
|
q |
Quit testing without running any more tests. |
|
d |
Enter the debugger at the point of failure. [Currently requires IPython] |
|
c |
Continue running tests. If the test failed due to an exception, testing resumes with the next test. If the problem was an assertion failure, testing continues within the same test. |
|
r |
Redo the same test from the beginning. |
|
e |
Jump emacs to the point of failure. [May not work in your environment, uses gnuserve. See ltest.runner.emacsto] |
The suite function accepts the following keyword arguments:
|
verbose |
True/false. The runner will report its progress. |
|
Select |
Only execute the specified test or test-group. E.g. select="network.errors" |
The following assertion operators are available:
|
<expr> ? |
Assert true |
|
<expr> ?! |
Assert false |
|
<expr> ?= <expr> |
Assert equal |
|
<expr> ?!= <expr> |
Assert not equal |
|
<expr> ?? <expr> |
Assert LHS matches RHS pattern |
|
<expr> ?raises <expr> |
Assert LHS raises an exception matching RHS pattern |
The last two operators use object-patterns, described in the next section.
Often, the subject of assertions are not simple values but complex objects, containing many attributes which may themselves be object structures. There are two common approaches to this.
Writing many assertions quickly gets tedious. Equality tests can be lengthy and are not very informative: when two objects are found to be not equal, there will be no indication why. The tester is often forced to use a post-mortem debugger to discover what part of the object state caused the inequality.
Logix-Test object patterns are an alternative which is often preferable. They provide two main advantages: they are very concise, and if a match fails, the pattern matching mechanism will report exactly where.
The pattern captures the expected type, attribute values, and ‘contents’ of an object (‘contents’ refers to the values returned by the subscript operator, or if you prefer, by the __getitem__ method).
We can explore the syntax by switching to the test-language at the Logix prompt.
[std]: limport ltest.testlang
[std]: setlang testlang.testlang
[testlang]:
A simple pattern that matches objects of type str looks like.
{:str}
Patterns are designed to be used within the context of a test, using the pattern assertion operator described above, e.g.:
someObject ?? {:str}
They can also be used at the prompt by calling the test method. It returns True if the pattern matches, or an diagnostic message if it does not.
[testlang]: {:str}.test 'a'
True
[testlang]: print {:str}.test 1
: wrong type:
1
Expected type: str
Found type : int
Note that subtypes will also match.
[testlang]: {:basestring}.test 'a'
True
Strict type matching can be specified by appending a ‘!’ to the type.
[testlang]: print {:basestring!}.test 'a'
: wrong type:
'a'
Expected type: str
Found type : basestring
The pattern can also include tests on object attributes.
[testlang]: class O:
[testlang]: o = O()
[testlang]: o.a=1
[testlang]: o.b=2
[testlang]: {:O a=1}.test o
True
[testlang]: print {:O a=2}.test o
.a: not equal:
2
---
1
[testlang]: {:O a=1 b=2}.test o
True
Note that the pattern is not an exhaustive description of the object. Any attributes not mentioned are simply not tested.
To simply test is a value is a true value:
[testlang]: {:O a?}.test o
True
[testlang]: o.a=0
[testlang]: print {:O a?}.test o
.a: 0 is not True
If you do not wish to specify a type, put * for the type.
[testlang]: {:* a=1 b=2}.test o
True
If the object supports the list-indexing operator (/[ ] in Standard Logix), the pattern can test the values returned for given keys. Currently only string and integer keys are supported.
Values that appear directly in the pattern are compared with keys 0, 1, 2 and so on.
[testlang]: l = [5, 6, 7, 8]
[testlang]: {:* 5 6 7 8}.test l
True
[testlang]: {:* 5 6}.test l
True
To test for a specific key, use / in front of the key.
[testlang]: {:* /3=8}
True
To test for all the numeric keys use /*
[testlang]: {:* /*=[5, 6, 7, 8]}.test
l
True
[testlang]: print {:* /*=[5, 6, 7]}.test l
: length mismatch:
[5, 6, 7]
---
[5, 6, 7, 8]
String keys can also be used. Names are automatically quoted.
[testlang]: d = dict foo=1 baa=2
[testlang]: {:* /foo=1}.test d
True
[testlang]: print {:* fo=1}.test d
: no such location /fo
Explicit quotes can also be used.
[testlang]: {:* /"foo"=1}.test d
True
The value with which a comparison is made can itself be a pattern
[testlang]: o.a = O()
[testlang]: o.a.x = 'ho'
[testlang]: o.a.y = 'hum'
[testlang]: {:O a={:O x='ho' y='hum'}}.test o
True
[testlang]: {:O a={:O x='ho' y='hummm'}}.test o
.a.y: not equal:
'hummm'
---
'hum'
Lists can also contain patterns
[testlang]: o.l = [1, [2, 3, 4], 5]
[testlang]: {:O l=[1, {:* /2=4}, 5]}.test o
True
So far we have tested only for equality. If you compare an attribute with a function, the function is called as a predicate.
[testlang]: {:O l={len it == 3}}.test o
True
[testlang]: print {:O l={len it == 2}}.test o
.l: test failed for:
[1, [2, 3, 4], 5]
If value being tested is a string, a regex can be used as the test value.
[testlang]: import re
[testlang]: o.s = "Ugh! So many features to document"
[testlang]: {:O s=/fe[a-z]t/}.test o
True
At the end of the pattern, an arbitrary test can be included after an &. The tested object is available as it. For example
{:Customer & it in preferred}
Mock-objects are a great idea for unit tests. They can make tests much more compact and maintainable. They make it much easier to isolate tests, i.e. to avoid accessing parts of the program that are not the subject of the test. They yield tests that ‘fail fast’. They help avoid making your code test aware. They make it much easier to test for error conditions (e.g. simulate a network error). (See www.mockobjects.com for more information).
However, mock-objects have their down side too. Mock-object frameworks tend to be rather limited in their facilities for expressing the ordering of expectations. Sometimes you may be forced to write over-constraining tests – say, requiring certain events to occur in sequence, when in fact all you care about is that they all occur once. On other occasions, it may be difficult to specify that expectation X must occur before expectation Y, particularly if they are expectations of different mock-objects.
Another down-side, is that mock-objects can be very tedious to write.
Logix-Test provides a mock-object framework that attempts to address these difficulties. It features a compact, declarative syntax for creating mock-objects and specifying expectations. It supports a flexible style of ordering expectations, which can be applied to both individual mock-objects, and whole groups.
The operator defmob creates a new mock-object. The mock is given a name and an expectation. Here is a simple mock called ‘foo’ that expects a single call to the method callme, and will return None to the caller:
defmob foo callme() -> None
The general form is
defmob <name> <expectation>
The expectation is written in a special mini-language. Here are some expectations in the language:
|
Expression |
Expectation |
|
turnRight() -> True |
Call to turnRight with no arguments. Returns True. |
|
accelerate 50 -> None |
Call to accelerate with argument 50. Returns None. |
|
speed = 100 |
Set attribute speed to 100. |
|
speed -> 50 |
Get attribute speed. Returns 50 |
|
/foo -> 0 |
Get item "foo". Returns 0 |
|
/(x) -> 0 |
Get item x (i.e. the key is the run-time value of variable x). Returns 0 |
|
/foo = 100 |
Expect item "foo" to be set to 100. [NOT IMPLEMENTED] |
|
seq: |
The given expectations in sequence |
|
<exp1> ; <exp1> |
(alternative syntax for seq) |
|
par: |
The given expectations in any order |
|
<exp1> || <exp2> |
(alternative syntax for par) |
|
<expectation>* |
Zero or more occurrences of the given expectation |
For example
defmob customer par:
name -> "I'm a mock customer" *
seq:
salary = 100
promoteTo 'manager'
-> True
salary -> 150
This mock object expects the salary attribute to be set to 100, followed by a call to promoteTo, and finally expected the salary attribute to be accessed. It also expects any number of accesses to the name attribute, arbitrarily interleaved with this sequence.
As seen, a method call expectation can contain expectations for the values passed as arguments. An attribute-set expectation contains an expectation for the value being set.
There are three ways in which these expected values can be expressed: By equal value, by predicate function and by object pattern. The default is that the expected value and the value actually passed should be equal. E.g.
promoteTo 'Manager' -> True
and
speed = 100
If however, you pass a function as the expected value, it will be used as a predicate to test the value actually passed.
speed = {it > 100}
If you do not wish to constrain the value, you could define a function like:
def any x: True
This can be used directly in expectations:
promoteTo any -> True
Take care if the expected value is itself a function. Say you expect the function f to be passed. The following expectation is incorrect:
callback = f
The function f will be interpreted as a predicate and called by the mock-object framework! Instead write
callback = {it==f}
Finally, if the expected value is an object pattern, the value actually passed will be tested against this pattern.
setSalesContact {:SalesExec rank=4} -> None
The purpose of a mock object is to simulate, for testing purposes, the environment that some piece of code expects to be in. The expected environment will often consist of multiple objects, and it is useful to be able to specify patterns of expectations in terms of the whole group. For example “expect a call to x.m before any calls to y.m”.
This is the purpose of the mobgroup operator.
mobgroup <name> <name> ... expecting <group-expectation>
mobgroup allows multiple mock-objects to be created, and a single expectation to be defined for the entire group. The expectation language is extended so that the traditional dot operator can be used to specify which object the expectation applies to. The / operator can also be used to specify item access on a particular mock-object.
For example:
mobgroup employee department expecting par:
department.name -> 'sales' *
seq:
department/"manager"
-> employee
par:
employee.salary -> 1000000 *
employee.name -> 'mock employee *
employee.fire() -> None
Sometimes, the code being tested will incorporate type-checks (such as isinstance). In this case, it may not function correctly with mock-objects. To overcome this you can create mock objects that extend a class of your choosing. All attribute access to the mock-object is intercepted, so the behavior will be unaffected.
To define a mock object with a base class using defmob:
defmob mockCustomer(Customer) ...
And using mobgroup:
mobgroup e(Employee) dpt(Department) ...
Mock objects will raise an exception if the client code does something unexpected. If however, the problem is that the client code did not do all the things that were expected, there may not be an exception raised. In other words, some tests will need to conclude by confirming that the pattern of expectations has completed.
To confirm an individual mock-object has received all of its expectations, call the confirmDone function from module testlang.
testlang.confirmDone myMob
To confirm a group expectation has fully completed, keep a reference to the group object, and call its confirmDone method.
g = mobgroup a b c expecting ...
... test actions
g.confirmDone()
The expectation language defines a postfix operator :debug which can be applied to any expectation. This causes the test to break into the debugger when that expectation is met.
defmob customer par:
name -> "I'm a mock customer" *
seq:
salary = 100
promoteTo 'manager'
-> True :debug
salary -> 150