Chapter 16: Testing
Where you will learn how to avoid annoying bug reports.
Automated testing is a fundamental aspect of modern software development.
Long gone are the days of manual, ad-hoc testing.
With modern testing tools and frameworks, it's (somewhat) easy to increase the quality of programs by implementing comprehensive test suites that can cover large percentages of a program's functionality and behavior.
Lux doesn't stay behind and includes a testing module as part of its standard library.
The lux/test module contains the machinery you need to write unit-testing suites for your programs.
Not only that, but the Leiningen plugin for Lux also includes a command for testing, in the form of lein lux test.
How do you set that up? Let's take a look at the project.clj file for the Lux standard library itself.
(defproject com.github.luxlang/lux-stdlib "0.5.0"
:description "Standard library for the Lux programming language."
:url "https://github.com/LuxLang/stdlib"
:license {:name "Mozilla Public License (Version 2.0)"
:url "https://www.mozilla.org/en-US/MPL/2.0/"}
:plugins [[com.github.luxlang/lein-luxc "0.5.0"]]
:deploy-repositories [["releases" {:url "https://oss.sonatype.org/service/local/staging/deploy/maven2/"
:creds :gpg}]
["snapshots" {:url "https://oss.sonatype.org/content/repositories/snapshots/"
:creds :gpg}]]
:pom-addition [:developers [:developer
[:name "Eduardo Julian"]
[:url "https://github.com/eduardoejp"]]]
:repositories [["snapshots" "https://oss.sonatype.org/content/repositories/snapshots/"]
["releases" "https://oss.sonatype.org/service/local/staging/deploy/maven2/"]]
:source-paths ["source"]
:test-paths ["test"]
:lux {:tests "tests"}
)
The :tests parameter is similar to the :program parameter in that it specifies the name of a module file (this time, inside the test directory).
Here are the contents of the file:
(;module:
lux
(lux (control monad)
(codata [io])
(concurrency [promise])
[cli #+ program:]
[test])
(test lux
(lux ["_;" cli]
["_;" host]
["_;" pipe]
["_;" lexer]
["_;" regex]
(codata ["_;" io]
["_;" env]
["_;" state]
["_;" cont]
(struct ["_;" stream]))
(concurrency ["_;" actor]
["_;" atom]
["_;" frp]
["_;" promise]
["_;" stm])
(control [effect])
(data [bit]
[bool]
[char]
[error]
[ident]
[identity]
[log]
[maybe]
[number]
[product]
[sum]
[text]
(error [exception])
(format [json])
(struct [array]
[dict]
[list]
[queue]
[set]
[stack]
[tree]
## [vector]
[zipper])
(text [format])
)
["_;" math]
(math ["_;" ratio]
["_;" complex]
## ["_;" random]
["_;" simple])
## ["_;" macro]
(macro ["_;" ast]
["_;" syntax]
(poly ["poly_;" eq]
["poly_;" text-encoder]
["poly_;" functor]))
["_;" type]
(type ["_;" check]
["_;" auto])
)
))
## [Program]
(program: args
(test;run))
This looks very weird.
There's almost nothing going on here, yet this is the most important file in the whole test suite (this is where everything comes together and the tests are run).
But where do those tests come from? Nothing is being defined here.
Well, the run macro, from lux/test pulls in all the tests from the imported modules to run them later once the program starts.
To know how tests work, let's take a look at one of those modules.
From
test/lux/concurrency/promise.
(;module:
lux
(lux (control monad)
(data [number]
text/format
[error #- fail])
(concurrency ["&" promise])
(codata function
[io #- run])
["R" random]
pipe)
lux/test)
(test: "Promises"
($_ seq
(do &;Monad<Promise>
[running? (&;future (io true))]
(assert "Can run IO actions in separate threads."
running?))
(do &;Monad<Promise>
[_ (&;wait +500)]
(assert "Can wait for a specified amount of time."
true))
(do &;Monad<Promise>
[[left right] (&;seq (&;future (io true))
(&;future (io false)))]
(assert "Can combine promises sequentially."
(and left (not right))))
(do &;Monad<Promise>
[?left (&;alt (&;delay +100 true)
(&;delay +200 false))
?right (&;alt (&;delay +200 true)
(&;delay +100 false))]
(assert "Can combine promises alternatively."
(case [?left ?right]
[(#;Left true) (#;Right false)]
true
_
false)))
(do &;Monad<Promise>
[?left (&;either (&;delay +100 true)
(&;delay +200 false))
?right (&;either (&;delay +200 true)
(&;delay +100 false))]
(assert "Can combine promises alternatively [Part 2]."
(and ?left (not ?right))))
(assert "Can poll a promise for its value."
(and (|> (&;poll (:: &;Monad<Promise> wrap true))
(case> (#;Some true) true _ false))
(|> (&;poll (&;delay +200 true))
(case> #;None true _ false))))
(assert "Cant re-resolve a resolved promise."
(and (not (io;run (&;resolve false (:: &;Monad<Promise> wrap true))))
(io;run (&;resolve true (: (&;Promise Bool) (&;promise))))))
(do &;Monad<Promise>
[?none (&;time-out +100 (&;delay +200 true))
?some (&;time-out +200 (&;delay +100 true))]
(assert "Can establish maximum waiting times for promises to be fulfilled."
(case [?none ?some]
[#;None (#;Some true)]
true
_
false)))
))
You define a test using the test: macro, which just takes a description instead of a name.
The seq function (from lux/test) allows you to combine tests sequentially so that one runs after the other, and if one fails, everything fails.
Also, all tests are assumed to be asynchronous and produce promises, which is why you see so much monadic code in the promise monad.
The assert function checks if a condition is true, and raises the given message as an error otherwise.
Oh, and there's also a type for tests:
(type: #export Test
{#;doc "Tests are asynchronous process which may fail."}
(Promise (Error Unit)))
There is also a more advanced way to write tests called property-based testing.
The idea is this: unit tests could also be called "testing by example" because you pick the particular inputs and outputs that you want to test you code against.
The problem is that you might miss some cases which could cause your tests to fail, or you might introduce some bias when picking cases, such that faulty code could still pass the tests.
With property-based testing, the inputs to your tests are generated randomly, and you're then forced to check that the invariants you expect are still kept.
The testing framework can then produce hundreds of variations, exposing your code to far more rigurous testing than unit tests could provide.
Lux already comes with great support for this, by producing random values thanks to the lux/random module, and integrating them in the tests with the help of the test: macro.
For example:
## From test/lux in the standard library's test-suite.
(;module:
lux
lux/test
(lux (control monad)
(codata [io])
[math]
["R" random]
(data [text "T/" Eq<Text>]
text/format)
[compiler]
(macro ["s" syntax #+ syntax:])))
(test: "Value identity."
[size (|> R;nat (:: @ map (|>. (n.% +100) (n.max +10))))
x (R;text size)
y (R;text size)]
($_ seq
(assert "Every value is identical to itself, and the 'id' function doesn't change values in any way."
(and (is x x)
(is x (id x))))
(assert "Values created separately can't be identical."
(not (is x y)))
))
By just adding bindings to your test, you're immediately working on the Random monad and can use any of the generators and combinators offered by the lux/random module.
By default, your test will be run 100 times, but you can configure how many times do you want with the #times option:
(test: "Some test..."
#times +12345
[... ...]
...
)
And if your test fails, you'll be shown the seed that the random number generator used to produce the failing inputs.
If you want to re-create the conditions that led to the failure, just use the #seed option:
(test: "Some test..."
#seed +67890
[... ...]
...
)
If you want to learn more about how to write tests, feel free to check out the test-suite for the Lux standard library. It's very comprehensive and filled with good examples. You can find it here: https://github.com/LuxLang/lux/tree/master/stdlib/test.
Without tests, the reliability of programs becomes a matter of faith, not engineering.
Automated tests can be integrated into processes of continuous delivery and integration to increase the confidence of individuals and teams that real value is being delivered, and that the customer won't be dissatisfied by buggy software.
Now that you know how to test your programs, you know everything you need to know to be a Lux programmer.