The Semantics of Testing. Also, quads.

[jjtolton]

Continued from PR on library(testing).

What are the semantics of testing in Prolog? What should they be?

What are "quads"? What role do they play in testing?

[UWN]

What are "quads"? What role do they play in testing?

Queries using answer descriptions (quads) are a notation to describe the answers of a query. They are used to describe examples, test cases and bug reports. They are essentially a transcript of a top-level interaction plus some additional features, like l#1:

?- length(L,N).
   L = [], N = 0
;  L = [_A], N = 1
;  L = [_A,_B], N = 2
;  ... .

So this is a test that ensures that the first three leaf answers look exactly like that (_A and _B must be uninstantiated and unaliased), and that there are no answer constraints.

Or consider ph#1:

1 ?- phrase(=,L).
     L = []
  |  existence_error(non_terminal,(=)//0).

So either there is one solution L = [] or there is an existence error for a non-terminal (=)//0.

Compared to test frameworks like plunit, quads have much less syntactic overhead - most of which is part of the top-level already - and fit into program text directly.

A query may also have more than one answer description. This is useful for bug reports like #2398

?- phrase(!,[]),false;true.
   false, unexpected.
   true. % expected, but not found

So the first answer description shows the current unexpected answer and adds the comment unexpected that it is unexpected. In this manner the bug report is transformed into a test case that needs no further editing even after fixing the problem. The second answer description shows the expected result. The comment is there just for clarity.

There are still quite a lot of open questions when it comes to more complex queries, in particular in the context of answer constraints. Hopefully, all this will sort out.

[UWN]

• Is the indentation significant?

No.

• How do we distinguish between a leaf answer in a quad and a new unrelated fact? Unless I'm mistaken, it seems like the portability of quads would be restricted to Prologs that have quads implemented, which I suppose is fine, I just want to make sure I'm understanding that correctly.

By the principal functor. Many of which are already protected against modifications, like (',')/2. This protection also serves to identify answer descriptions without a preceding query. Of course this means that the number of principal functors should be kept low.

[jjtolton]

So in this case, we are protected by the fact that the definition of (=)/2 is unalterable?

[UWN]

So in this case, we are protected by the fact that the definition of (=)/2 is unalterable?

Exactly.

[bakaq]

How would it work with residual goals?

?- dif(X,Y).
   dif:dif(X,Y).

I think in this case there is no problem because the predicate dif/2 in the module dif is not dynamic or multifile, so it is also protected. Do we just assume that any residual goal will also be protected?

[UWN]

How would it work with residual goals?

It could work as you suggest. Still unsatisfactory when the precise representation of answer constraints changes. For the moment, just a maybe for constraints (like in the original CLP(R)) might be the best first step.