diff --git a/MANIFEST b/MANIFEST index 90a3b5d22..92ee2fd27 100644 --- a/MANIFEST +++ b/MANIFEST @@ -1,8 +1,6 @@ Changes CONTRIBUTING.md -docs/Implementing_Tests.pod docs/Translation-translators.md -docs/Translation.pod inc/Module/Install.pm inc/Module/Install/Base.pm inc/Module/Install/Can.pm @@ -19,6 +17,7 @@ lib/Zonemaster/Engine/ASNLookup.pm lib/Zonemaster/Engine/Constants.pm lib/Zonemaster/Engine/DNSName.pm lib/Zonemaster/Engine/Exception.pm +lib/Zonemaster/Engine/Localization.pod lib/Zonemaster/Engine/Logger.pm lib/Zonemaster/Engine/Logger/Entry.pm lib/Zonemaster/Engine/Nameserver.pm @@ -44,6 +43,7 @@ lib/Zonemaster/Engine/Test/Syntax.pm lib/Zonemaster/Engine/Test/Zone.pm lib/Zonemaster/Engine/TestMethods.pm lib/Zonemaster/Engine/TestMethodsV2.pm +lib/Zonemaster/Engine/TestModuleInterface.pod lib/Zonemaster/Engine/Translator.pm lib/Zonemaster/Engine/Util.pm lib/Zonemaster/Engine/Zone.pm diff --git a/docs/Implementing_Tests.pod b/docs/Implementing_Tests.pod deleted file mode 100644 index e9eaf0201..000000000 --- a/docs/Implementing_Tests.pod +++ /dev/null @@ -1,71 +0,0 @@ -=head1 INTRODUCTION - -The Zonemaster system intends to provide a framework for implementing DNS tests. In order to do this, it exposes APIs to the test code for examining a -DNS zone, to emit examination results and to access configuration information on how the tests should be performed. - -=head1 RULES AND RECOMMENDATIONS FOR TEST CODE - -=over - -=item Only access the outside world via the framework APIs. - -The framework is built to make test results traceable, repeatable and understandable. If the test code itself goes around the framework and accesses -things on its own, all those attributes are lost. If the test you're writing needs something the framework doesn't provide, request that it be added -to the framework. - -=item Make as few assumptions as possible. - -When the entry method in a test module is called, it can only rely on a handful of things about the testing process so far: - -=over - -=item - -The zone object it's given has a parent. - -=item - -The zone object it's given has glue records. - -=item - -The zone object it's given has nameservers. - -=back - -=item Favor code clarity over performance. - -Strive to make the test code as easily understandable as possible. Ideally, it should be possible for a person with only a basic understanding of Perl to -read the code and verify that it correctly implements the corresponding Test Case specifications. - -=back - -=head1 TEST MODULE STRUCTURE - -The Zonemaster framework will attempt to load and call all modules in the namespace C. These modules must have three class methods -defined. One to run all tests it provides, and two to expose metadata about the module. Apart from that (and the rules above) the module can be -implemented as desired. - -=head2 Metadata methods - -=over - -=item C - -This method should return a string representing the current version of the test module. The version information will be logged in a C -message, but otherwise not used. - -=item C - -This method should return a reference to a hash. The keys of this hash should be the names of the individual test methods in the module, and the -values for each key should be a reference to an array listing all the possible messages that test methods can emit. - -=back - -=head2 Test-running method(s) - -The framework expects to be able to run all tests in a module by calling a class method L with a -L object as the argument. -The return value from this method is expected to be a list of L objects describing the results of the tests run. - -=cut diff --git a/docs/Translation.pod b/lib/Zonemaster/Engine/Localization.pod similarity index 51% rename from docs/Translation.pod rename to lib/Zonemaster/Engine/Localization.pod index f0b9191c7..f7202d409 100644 --- a/docs/Translation.pod +++ b/lib/Zonemaster/Engine/Localization.pod @@ -1,6 +1,8 @@ -=head1 TRANSLATION +=head1 NAME -=head2 Introduction +Zonemaster::Engine::Localization - On L. + +=head1 DESCRIPTION The translation system in Zonemaster is a two-step process, where internal message tags are first replaced by English strings with argument @@ -11,41 +13,17 @@ All translation files live in the F directory in the L source directory and all commands described here are executed from that directory. -=head2 For translators +=head1 FOR TRANSLATORS Instructions for translators can be found at L -=head2 For developers of Zonemaster test modules - -The test module code should produce log messages with message tags, as documented -elsewhere. These tags will be used for translation to human language, for -determining the severity of the event logged and to make the events easily used -by other software. - -Each test module must also have a method named C. -This method must return a reference to a hash whose entries are expected to look -like this, where C is a message, C is the name of a -test module tag and C<"Hello, {name}!"> is a message id: - - MESSAGE_TAG => sub { - __x # TEST_MODULE:MESSAGE_TAG - "Hello, {name}!", @_; - }, - -A number of things are important here. -Keys in the hashref are message tags and values are coderefs. -The coderef calls Locale::TextDomain::__x() with a Perl brace format string -(a.k.a. message id) and passes along its own @_). -The coderef propagates the return value of Locale::TextDomain::__x(). -The line immediately before the format string contains a comment consisting of -the module name, a colon and the message tag. +=head1 FOR DEVELOPERS -The format strings themselves, the comments and the line numbers of the __x -calls are used by the gettext tooling when updating the PO files with new -message ids and old message strings. +Documentation for developers of Zonemaster test modules can be found at +L. -=head2 For Zonemaster package maintainers +=head1 FOR MAINTAINERS In order to make a new translation usable, it must be compiled to C format and installed. The first step needs the C program from the GNU gettext diff --git a/lib/Zonemaster/Engine/Profile.pm b/lib/Zonemaster/Engine/Profile.pm index 8545d8211..2b6810c6a 100644 --- a/lib/Zonemaster/Engine/Profile.pm +++ b/lib/Zonemaster/Engine/Profile.pm @@ -905,7 +905,7 @@ An arrayref of names of implemented test cases (in all lower-case) as listed in L. Default is an arrayref listing all the test cases. -Specifies which test cases can be run by the testing suite. +Specifies which test cases can be run by the test harness. Note that an exception applies to test cases C and C: when running either the full testing suite or just the Basic test module, diff --git a/lib/Zonemaster/Engine/Test.pm b/lib/Zonemaster/Engine/Test.pm index f68c84841..b5ab1af48 100644 --- a/lib/Zonemaster/Engine/Test.pm +++ b/lib/Zonemaster/Engine/Test.pm @@ -39,52 +39,7 @@ Zonemaster::Engine::Test - Module implementing methods to find, load and execute =head1 TEST MODULES Test modules are defined as modules with names starting with C. -They are expected to provide at least the following class methods: - -=over - -=item all() - -This will be given a L object as its only argument, and, after running the -Test Cases for that Test module, is expected to return a list of L objects. -This is the entry point used by the L and L methods of this class. - -=back - -=cut - -=over - -=item version() - -This must return the version of the Test module. - -=back - -=cut - -=over - -=item metadata() - -This must return a reference to a hash, the keys of which are the names of all Test Cases in -the module, and the corresponding values are references to an array containing all the message -tags that the Test Case can use in L. - -=back - -=cut - -=over - -=item tag_descriptions() - -This must return a reference to a hash, the keys of which are the message tags and the corresponding values -are strings (message IDs) corresponding to user-friendly English translations of those message tags. -Keep in mind that the message ids will be used as keys to look up translations into other languages, -so think twice before editing them. - -=back +They are expected to implement the L. =cut diff --git a/lib/Zonemaster/Engine/TestModuleInterface.pod b/lib/Zonemaster/Engine/TestModuleInterface.pod new file mode 100644 index 000000000..c6de1b1cc --- /dev/null +++ b/lib/Zonemaster/Engine/TestModuleInterface.pod @@ -0,0 +1,169 @@ +=head1 NAME + +Zonemaster::Engine::TestModuleInterface - On the test module interface. + +=head1 BACKGROUND + +The Zonemaster system provides a harness for running DNS tests and facilities for +implementing them. + +The harness requires that the tests are organized into test modules and further +into test cases. +It loads all built-in test modules and exposes APIs to run their constituent +test cases. + +The facilities for implementing DNS tests include APIs to examine a DNS zone, to +access configuration information on how the tests should be performed and to +emit examination results. + +The examination results are structured as a list of log entries with message +tags and named arguments. +The message tags are used for determining the severity of the entry, for +translation to human language and to make the examination results easily used by +other software. + +=head1 RULES AND RECOMMENDATIONS + +=head2 Accessing the outside world + +Only access the outside world via the framework APIs. +The framework is built to make test results traceable, repeatable and +understandable. +If the test code itself goes around the framework and accesses things on its +own, all those attributes are lost. +If the test you're writing needs something the framework doesn't provide, +request that it be added to the framework. + +=head2 Make few assumptions + +Make as few assumptions as possible when implementing test modules. +When the entry method in a test module is called, it can only rely on a handful +of things about the testing process so far: + +=over + +=item + +The zone object it's given has a parent. + +=item + +The zone object it's given has glue records. + +=item + +The zone object it's given has nameservers. + +=back + +=head2 Code clarity over performance + +Strive to make the test code as easily understandable as possible. +Ideally, it should be possible for a person with only a basic understanding of +Perl to read the code and verify that it correctly implements the corresponding +Test Case specifications. + +=head2 Internationalization + +The Zonemaster system provides tooling for rendering message tags to human +language. +For this tooling to find the message tags the source code needs to be structured +in a certain way. +Each test module should contain a hash with one entry like this for each +message tag that should be renderable to human language, where C is +a message, C is the name of a test module tag and C<"Hello, +{name}!"> is a message ID. + + MESSAGE_TAG => sub { + __x # TEST_MODULE:MESSAGE_TAG + "Hello, {name}!", @_; + }, + +A number of things are important here. +The coderefs call L with a Perl brace format string (a.k.a. message ID) and propagates +its own C<@_>). +The coderefs propagate the return value from __x. +The line immediately before the format string contains a comment consisting of +the module name, a colon and the message tag. + +The message IDs should be expressed in user-friendly English translations of +their corresponding message tags. +Keep in mind that the message ids will be used as keys to look up translations +into other languages, so think twice before editing them. + +The format strings themselves, the comments and the line numbers of the __x +calls are used by the gettext tooling when updating the PO files with new +message IDs and old message strings. + +See also L. + +=head1 TEST MODULE INTERFACE + +A test module is a Perl module that implements the interface described here. + +The name of the test module is the last component of the package of the Perl +module. E.g. L implements a test module named +C. + +=head2 CLASS METHODS + +=head3 all() + +Runs the subset of the L that are configured in the +L of the +L. + +Accepts a L instance representing the zone to test. + +Returns a list of L objects. +The objects are an aggregate of what is returned by the test case methods. +all() may optionally include additional entries of its own. + +In case there are dependencies between the test cases, all() may skip one test +case method depending on what log entries were emitted from another. + +=head3 metadata() + +Returns a hashref mapping test case names to arrayrefs of message tags. + +The set of hash keys declares the set of test cases implemented by this test +module. +Each hash value should be an arrayref listing all the possible message tags +emitted by the corresponding L. + +The name of each test case must be the name of the test module in lowercase +followed by one or more digits. The message tags must be strings in +SCREAMING_SNAKE_CASE. + +=head3 tag_descriptions() + +Returns a hashref mapping message tags to translation function coderefs. + +The hash keys should be a subset of message tags that could be returned by +L. +Message tags omitted from this hash will not be rendered to human language. + +Each translation function accepts a hash of named arguments, namely the +ones emitted by the test module for the respective message tag. +Each translation function returns a human readable string. + +See L for additional requirements on the implementation. + +=head3 version() + +Returns a string representing the current version of the test module. +The version information will be emitted in a C log entry, but otherwise +not used. + +=head4 TEST CASE METHODS + +For each test case declared by L there must be a corresponding +class method. +This is the set of test case methods. + +Test case methods must accept a L object representing +the zone to test, and they return their examination results as a list of +L objects. + +=cut