Warning This step is ahead of the current version of [learn-ocaml].
There are 2 main grading functions:
test_function_<nb_args>_againt_solution: the usual. Test the
student code against a given solution written in the solution.ml
file.
test_function_<nb_args>: compare the student code to a series of
tests where both inputs and expected outputs are given. Note that
you still need to write a solution in solution.ml to build your
exercise session since learn-ocaml build tests your grader with
solution.ml as the student copy.
Grading functions for functions return a global report concatening 4 specific ones:
report resulting of comparison between student and solution
outputs. The function ~test is used to build this report. By
default, ~test uses the structural equality to compare outputs.
report resulting of comparison between student and solution standard
outputs. The function ~test_stdout is used to build this
report. By default, ~test_stdout is set to ignore standard output
and returns an empty report.
report resulting of comparison between student and solution standard
errors. The function ~test_stderr is used to build this report.
By default, ~test_stderr is set to ignore standard error and
returns an empty report.
report resulting of the result of function ~after and returns an
empty report by default.
test_function_<nb_args>_against_solutionval test_function_1_against_solution :
?gen: int ->
?test: 'b tester ->
?test_stdout: io_tester ->
?test_stderr: io_tester ->
?before_reference : ('a -> unit) ->
?before_user : ('a -> unit) ->
?after : ('a -> ('b * string * string) -> ('b * string * string) -> Learnocaml_report.report) ->
?sampler : (unit -> 'a) ->
('a -> 'b) Ty.ty -> string -> 'a list -> Learnocaml_report.item
test_function_1_againt_solution_1 ty name tests]:ty: type of the tested function specified for the given tests. It
must not contain type variables (i.e. 'a, 'b etc..), match the
type of the tests and be compatible with the solution.
name: name of the tested function in the student code and in
solution.ml
tests: list of tested inputs.
test_function_<nb_args>val test_function_1 :
?test: 'b tester ->
?test_stdout: io_tester ->
?test_stderr: io_tester ->
?before : ('a -> unit) ->
?after : ('a -> ('b * string * string) -> ('b * string * string) -> Learnocaml_report.report) ->
('a -> 'b) Ty.ty -> string -> ('a * 'b * string * string) list -> Learnocaml_report.item
test_function_1 ty name tests]:ty: type of the tested function specified for the given tests. It
must not contain type variables (i.e. 'a, 'b etc..), match the
type of the tests and be compatible with the solution.
name: name of the tested function in the student code and in
solution.ml
(in, out, stderr, stdout) :: _ as tests: list of tuples where
the first element in is the argument passed to the tested function
(for a three-arguments functions tests will instead matched (in1,
in2, in3, out, stderr, stdout)). The second element out is the
expected output. stderr and stdout are the expected strings in
standard output and error output respectively.
gen: number of automatically generated tests. See
step-3
and
step-4
for more information.
sampler: used to define sampler for automatically generating
inputs for tests. See
step-3
and
step-4
for more information.
test: is used to redefine the function which compare the output of
the student function and the output of the solution. See WIP for
more information.
test_stdout: is used to redefine the function which compare the
standard output channel of the student function and the one of the
solution. See WIP for more information.
test_stderr: is used to redefine the function which compare the
standard error channel of the student function and the one of the
solution. See WIP for more information.
after: is used to redefine a function which is called with the
current tested inputs, the student result and the solution result
and returns a report. Enables for example to inspect references
introduced with ~before, ~before_user or ~before_reference and
build an appropriate report. See WIP for more information.
before_reference: is used to redefine a function called right
before the application of the current tested inputs to the
solution. This enables for example to introduce a reference or make
a side effect before each test. See WIP for more information.
before_user: is used to redefine a function called right before
the application the current tested inputs to the student
function. This enables to introduce a reference or make a side
effect between solution evaluation and student function
evaluation. See WIP for more information.
before: same as before_reference for
test_function_<nb_args>. Since no solution is evaluated, there is
no need to distinguish between before or after solution evaluation.
Note: only trivial examples can be found here. For more advanced examples, see the corresponding tutorials.
This is a classical example of an unary function, with a user-defined sampler.
let exercise_1 =
test_function_1_against_solution
[%ty: int -> int]
"identity"
~gen:10
~sampler:(fun () -> Random.int 42)
[0 ; 42]
With test_function_1:
let exercise_2 =
test_function_1
[%ty: int -> int]
"identity"
[0, 0, "", "" ;
42, 42, "", ""] (* List of tests *)
Note that since the default comparison function for standard output and error actually ignore them, we could put anything as expected outputs in the tests list.
Hello world is also very classical, however grading this function is
a little more tricky since by default standard output and error output
are ignored (meaning the student standard/error output can be
different from the solution ones without causing a failure).
let exercise_2 =
test_function_1_against_solution
[%ty: unit -> unit]
~test:test_ignore
~test_stdout:io_test_equals
"hello"
~gen:0
[()]
let exercise_4 =
test_function_1
[%ty: unit -> unit]
~test:test_ignore
~test_stdout:io_test_equals
"hello"
[(), (), "Hello world!", ""]
The function test_ignore for optional argument test enables to
ignore the student and solution output. On the contrary,
io_test_equals for test_stdout enables to compare standard outputs
with the structural equality.