For getting started with Testo and discovering the basics, make sure to read the tutorial first.
A simple solution is to use the built-in assert
construct:
let test_addition =
Testo.create "addition"
(fun () ->
let res = 1 + 1 in
assert (res = 2);
let res = 2 + 2 in
assert (res = 4)
)When such a test fails, the location of the assert ...
expression is shown to the user. The advantage over using a dedicated
function like Alcotest.check is that it’s a little quicker
to write and simpler to understand. The main disadvantage is that the
expected and actual values are not printed. In this case, it’s best for
the test to print enough information about the condition being tested
e.g.
open Printf
let test_addition =
Testo.create "addition"
(fun () ->
let add a b expected_result =
let res = a + b in
printf "checking %i + %i, expecting %i, result is %i.\n"
a b expected_result res;
assert (res = expected_result)
in
add 1 1 2;
add 2 2 4
)In case of a failure, the test’s output will be shown to the user.
Testo was originally meant to extend Alcotest, but it became too different and ended up being a separate project. However, Alcotest provides some functionality that wasn’t reimplemented in Testo and can benefit Testo tests. Like Testo, Alcotest is an ordinary OCaml library that is typically installed with Opam:
$ opam install alcotestAlcotest.check is the function we’ll use to check test
results against expectations. If a check fails, an exception is raised
and it is formatted as a nice error message.
Here’s a test that checks values of type int:
let test_addition =
Testo.create "addition"
(fun () ->
let res = 1 + 1 in
Alcotest.check Alcotest.int "sum" 2 res;
let res = 2 + 2 in
Alcotest.check Alcotest.int "sum" 4 res
)It is more compactly written as:
let test_addition =
Testo.create "addition"
(fun () ->
let res = 1 + 1 in
Alcotest.(check int) "sum" 2 res;
let res = 2 + 2 in
Alcotest.(check int) "sum" 4 res
)Alcotest.int is called a “testable”.
There are predefined testables for OCaml’s simple types such as
bool, int, string, etc.
Checking a list of strings can be done as follows:
let test_items =
Testo.create "items"
(fun () ->
let res = ["a"] @ ["b"] in
Alcotest.(check (list string)) "sum" ["a"; "b"] res
)A testable (see previous section) must be created for types such as
records or variants. This is done with
Alcotest.testable print equal where print is a
printer and equal is an equality function.
Let’s assume we’re testing values of type Thing.t.
Module Thing exposes the following signature:
module Thing : sig
type t
val to_string : t -> string
val compare : t -> t -> int
...
endThe test program will define a testable as follows:
let print_thing fmt x = Format.pp_print_string fmt (Thing.to_string x)
let equal_thing a b = (Thing.compare a b = 0)
let thing = Alcotest.testable print_thing equal_thingA test function will call Alcotest.check as follows:
let test_things =
Testo.create
"things"
(fun () ->
let result = ... in
let expected_thing = ... in
Alcotest.check thing "equal" expected_thing result
)There are two ways. If you want to match the output exactly and
review any differences conveniently, you should use the built-in
snapshotting mechanism. It is sufficient to pass the
~checked_output:(Testo.stdout ()) option to
Testo.create to check stdout. Similarly, you can check
stderr alone, stderr combined with stdout, or check stderr separately
from stdout (see API
reference).
let print_welcome_message () =
(* Typically, we would print something more complicated *)
print_endline ("hello " ^ Unix.getenv "USER")
let test_welcome_message =
Testo.create "welcome message"
~checked_output:(Testo.stdout ())
print_welcome_messageWhen running the test, any line difference would be highlighted and
would make the test fail. It is then your responsibility to either fix
the source code until the test passes or to approve the new output by
running ./test approve -s 'welcome message'. This saves the
expected output as a snapshot file that you must track with Git or your
favorite VCS.
A common problem is that the output contains variable parts such as
timestamps or random identifiers, causing the test to fail from one run
to another or from one host to another. In our example, we want to hide
the user name taken from the USER environment variable. It
can be masked using masking functions provided by the Testo
module or by any function that takes a string and returns a new string.
Check out the API for the list
of built-in mask_* functions. Multiple masking
functions can be applied successively if needed. You must pass them as a
list to Testo.create as the ~normalize
argument. In the following example, we replace anything that follows
hello by <MASKED>:
let test_welcome_message =
Testo.create "welcome message"
~checked_output:(Testo.stdout ())
~normalize:[Testo.mask_line ~after:"hello " ()]
print_welcome_messageIf masking is too cumbersome or too brittle, a better approach is to
only check for the presence of this or that substring or pattern. Here,
we capture the standard output of our print_welcome_message
function directly as out_str and then we check that it
looks right:
let test_welcome_message =
Testo.create "welcome message"
(fun () ->
let _res, out_str = Testo.with_capture stdout print_welcome_message in
if not (Testo.contains_substring ~sub:"hello" out_str) then
failwith "standard output doesn't contain 'hello'"
)In this case, failwith raises an exception to signal the
test failure in the usual fashion. Note that
Testo.with_capture is an ordinary function that we could
outside of Testo to capture stdout or stderr. To capture stdout and
stderr, use two calls to Testo.with_capture:
let test_welcome_message =
Testo.create "welcome message"
(fun () ->
(* capture stderr (outer wrapper) and stdout (inner wrapper) *)
let out_str, err_str =
Testo.with_capture stderr (fun () ->
let (), out_str =
Testo.with_capture stdout print_welcome_message in
out_str
)
in
(* check out_str and err_str *)
if not (Testo.contains_substring ~sub:"hello" out_str) then
failwith "standard output doesn't contain 'hello'";
if err_str <> "" then
failwith ("unexpected error output: " ^ err_str)
)Testo supports two ways of writing tests that are known to fail. If
the test consistently fails in all environments, the recommended
solution is to use
~expected_outcome:(Should_fail "<insert excuse>"):
Testo.create
"my test"
~expected_outcome:(Should_fail "TODO")
(fun () ->
(* code that raises an exception *)
...
)If such a test raises an exception as expected, its status will be
shown as XFAIL and considered successful.
Now, if a test is “flaky” i.e. it sometimes fails and sometimes
succeed, it can be marked as “skipped”. The only difference with a test
that’s completely removed from the test suite is that it’s still listed
by ./test status -a and marked as SKIP instead
of being invisible and forgotten. In this example, we’ll skip a test
only if the platform is Windows:
let is_windows =
Sys.os_type = "Win32"
let test_something =
(* We don't run this test on Windows because <reasons> *)
Testo.create
"something"
~skipped:is_windows
(fun () -> ...)For example, the tests may need to read a configuration file whose path isn’t fixed, or maybe we want to see what happens when we change some settings.
One solution is to set an environment variable in the shell or parent process of the test program. However, environment variables are not great for the following reasons:
To avoid these issues and make parameters explicit, Testo provides a
--env (or -e) option that allows passing
key/value pairs when invoking the test program. Consult
--help for guidance:
$ ./test --help
...
-e KEY=VALUE, --env=KEY=VALUE
Pass a key/value pair to the function that creates the test suite.
KEY must be an alphanumeric identifier of the form
[A-Za-z_][A-Za-z_0-9]*. VALUE can be any string. This mechanism for
passing arbitrary runtime settings to the test suite is offered as
a safer alternative to environment variables.Suppose we want to pass the path etc/special-foo.conf to
the test program. We would invoke it as follows:
$ ./test run --env conf=etc/special-foo.conf
...The part of the OCaml program that produces the list of tests would
consult the conf variable as follows:
let tests env : Testo.t list =
let config_path =
match List.assoc_opt "conf" env with
| None -> "foo.conf" (* default *)
| Some path -> path
in
[
Testo.create ...;
...
]
let () =
Testo.interpret_argv
~project_name:"foo"
testsPassing multiple key/value pairs is done with multiple
-e or --env options:
$ ./test run -e conf=etc/special-foo.conf -e foo=bar
...The simplest way is to start and stop the event loop within each
test. With Lwt, this is normally done with
Lwt_main.run:
(* Generic higher-level function that converts a asynchronous
computation into a synchronous one. *)
let sync (func : unit -> unit Lwt.t) =
fun () ->
Lwt_main.run func
let test_sleep () : unit Lwt.t =
Lwt_unix.sleep 0.05
let tests = [
Testo.create "sleep" (sync test_sleep);
]On some platforms such as a JavaScript runtime, there is no
equivalent of Lwt_main.run. For these cases, we provide the
library testo-lwt. It exposes a Testo_lwt
module whose interface is almost identical to Testo
except that test functions have type unit -> unit Lwt.t
instead of unit -> unit.
For the Async library, we don’t provide testo-async but
it should be straightforward to add. Check the status of the corresponding GitHub
issue.
By default, there’s nothing special to do because the test executable
created by Testo will detect the number of CPUs available on the machine
and run the tests in parallel accordingly. If this is not satisfying,
use the -j option to use a specific number of workers:
$ ./test -j4If some tests fail because they can’t run in parallel, use the
-j0 option to run tests sequentially without creating a
worker process. Creating a single worker process with -j1
should also work for the purpose of running the tests sequentially.
If the test suite should run sequentially most of the time, you
should make it the default by specifying ~default_workers
in your OCaml test program:
let () =
Testo.interpret_argv
~project_name:"my project"
~default_workers:(Some 0)
(fun _env -> tests)If you have the option of running the test suite on multiple hosts in
parallel, you have to run a different ./test command on
each host. To partition the work into 4 parts, run
./test --slice 1/4, ./test --slice 2/4,
./test --slice 3/4, and ./test --slice 4/4
each on a different host. Check with your CI provider how to achieve
this conveniently.
By default, each test function can take as long as it needs. If it
enters an infinite loop or is blocked on some network operation, this
can block the whole test suite. To prevent this, tests can be configured
to time out after a while. This is done by setting the
max_duration option, which is a time limit in seconds. For
example, the following code applies a 10-second limit to all the tests
of a suite:
let tests =
List.map (fun test -> Testo.update ~max_duration:(Some 10.) test) testsTo avoid overriding the time limit for the tests that already have one, inspect each test before optionally updating it:
let tests =
List.map (fun (test : Testo.t) ->
match test.max_duration with
| None -> Testo.update ~max_duration:(Some 10.) test
| Some _ -> test
) testsIf a test times out, it will be reported as failed. Here’s the output for a test named “taking too long” designed to trigger a timeout after 10 seconds:
$ ./test status -l
...
┌──────────────────────────────────────────────────────────────────────────────┐
│ [FAIL] 9f23c62e5bdb taking too long │
└──────────────────────────────────────────────────────────────────────────────┘
• Path to captured log: _build/testo/status/test/9f23c62e5bdb/log
• Timed out. Current time limit: 10 seconds
• Log (stdout, stderr) is empty.
...Check for known issues at https://github.com/semgrep/testo/issues.
If your problem seems new, please file a new issue as a first step
toward its resolution. The --debug flag might reveal what’s
going on and can be helpful for debugging certain problems.
🚧 not implemented, see Issue #14.