ENH: add ParserKlass key to DTConfig; check xdoctest.DocTestParser

[ev-br]

Add plug-in ParserKlass key, illustrate on xdoctest's parser. Cross-ref #59

[ev-br]

The PR is green on CI because xdoctest is not installed (yet). Locally, with xdoctest installed, the xdoctest.parser.DoctestParser tests fails to find examples:

(Pdb) print(module_cases.func3.__doc__)

    Check that printed arrays are checked with atol/rtol
    >>> import numpy as np
    >>> a = np.array([1, 2, 3, 4]) / 3
    >>> print(a)
    [0.33  0.66  1  1.33]
    
(Pdb) p len(tests)
1
(Pdb) p len(tests[0].examples)
0

I wonder if https://github.com/scipy/scipy_doctest/pull/174/files#diff-75fda68530c6a37e0fa0d24cc9a00b0c40093965801a2d9eda1917a2e2567c27R118 is the correct way to use xdoctest's Parser, would you mind weighing in @Erotemic ?

[Erotemic]

That API is an intermediate step which returns a list of doctest parts (as well as non-doctest strings in the docstr). The return type of DoctestParser.parse is annotated incorrectly (something I just fixed as well as improving the example in this function). It should be List[xdoctest.doctest_part.DoctestPart | str], which is somewhat compatible with the stdlib parser, but it chunks the doctests in a different way because of its AST parsing. It doesn't require the REPL style execution that the stdlib doctest does.

Here is a MWE illustrating what xdoctest can do with the above test:

def mwe():
    docstr = """
        Check that printed arrays are checked with atol/rtol
        >>> import numpy as np
        >>> a = np.array([1, 2, 3, 4]) / 3
        >>> print(a)
        [0.33  0.66  1  1.33]
    """
    import ubelt as ub
    import xdoctest
    from xdoctest.doctest_example import DocTest as XDoctest

    # With the core API (removes surrounding string and just returns completed
    # doctest objects)
    doctests: list[XDoctest] = list(xdoctest.core.parse_docstr_examples(docstr))
    for doctest in doctests:
        result = doctest.run(on_error='return')
        print(f'result = {ub.urepr(result, nl=1)}')

    # With the Doctest Part API (similar to the stdlib API, but can group
    # contiguous statements without a "got/want" structure)
    from xdoctest.doctest_part import DoctestPart as XDoctestPart
    xparser = xdoctest.parser.DoctestParser()
    xparts: list[str | XDoctestPart] = xparser.parse(docstr)
    print(f'xparts = {ub.urepr(xparts, nl=1)}')

    # With stdlib API
    import doctest
    stdparser = doctest.DocTestParser()
    stdparts = stdparser.parse(docstr)
    print(f'stdparts = {ub.urepr(stdparts, nl=1)}')


if __name__ == '__main__':
    mwe()

This prints:

* DOCTEST : <modpath?>::<callname?>:0, line 3 <- wrt source file
* FAILURE: <modpath?>::<callname?>:0
result = {
    'exc_info': (<class 'xdoctest.checker.GotWantException'>, GotWantException('got differs with doctest want'), <traceback object at 0x75274efbdac0>),
    'passed': False,
    'skipped': False,
    'failed': True,
}
xparts = [
    '\nCheck that printed arrays are checked with atol/rtol',
    <DoctestPart(ln 2, src="import n...", want=None) at 0x75274efdef90>,
    <DoctestPart(ln 4, src="print(a)...", want="[0.33  0...") at 0x75274efdc250>,
]
stdparts = [
    '\nCheck that printed arrays are checked with atol/rtol\n',
    <doctest.Example object at 0x75274efde690>,
    '',
    <doctest.Example object at 0x75274efdf090>,
    '',
    <doctest.Example object at 0x75274efddf10>,
    '',
]

[ev-br]

Thanks @Erotemic ! I've updated the drop-in test to account for the difference in doctest.Example vs xdoctest.doctest_parts.DoctestPart in 235a51e, does this look more reasonable now?

(Pdb) p [x for x in dir(xparts[1]) if not x.startswith('_')]
['check', 'compilable_source', 'compile_mode', 'directives', 'exec_lines', 'format_part', 'has_any_code', 'line_offset', 'n_exec_lines', 'n_lines', 'n_want_lines', 'orig_lines', 'partno', 'source', 'want', 'want_lines']

(Pdb) p [x for x in dir(stdparts[1]) if not x.startswith('_')]
['exc_msg', 'indent', 'lineno', 'options', 'source', 'want']

One thing to note is that DoctestPart does not have options and indent attributes. Don't know if not having them is going to cause any issues down the line.

[Erotemic]

One thing to note is that DoctestPart does not have options and indent attributes. Don't know if not having them is going to cause any issues down the line.

Maybe, but maybe not for those reasons. Doctest parts cannot be run independently. They just tell you the lines of code to run (and what directives they will apply), but they have no context. Unlike stdlib doctest, xdoctest maintains a state as it executes a sequence of "examples / parts". Maybe this is ok, as the stdlib doctest runner needs to do something similar, as long as you don't do anything you wouldn't be able to do with regular doctest.

Note: if you instantiate the parser via DoctestParser(simulate_repl=True), you will get something closer to stdlib Examples in that it will try to make each "Part" a single statement.

indent attributes

That should be completely irrelevant. Everything should already be dedented based on the otput of the parser. I'm really not sure why stdlib doctests need this at all.

options attributes

In xdoctest these are maintained only at the xdoctest.doctest_example.DocTest level, which holds the list of DoctestParts. Unlike stdlib directives can be applied to multiple lines of code (by putting them on their own line, which means they apply to everything below it). So it doesn't make sense to store them at the "part" level because the options of a part will depend on the context in which they are run.

Something else to note is that xdoctest doesn't really have a concept of a "runner" as it is built into the run method of xdoctest.doctest_example.DocTest itself. Perhaps it could be factored into two classes for modularity, but the design is intended to only mimic the behavior of stdlib doctests and not necessarily its API.