API method get_test_results fails if version.bind string contains non-ASCII bytes

[marc-vanderwal]

Summary

When testing a domain, if any name server returns non-ASCII bytes in its response to version.bind/TXT/CH or version.server/TXT/CH queries, the test results fail to show on the GUI.

Description

Among the tests that Zonemaster carries out, there is Nameserver15, which checks if version.bind/TXT/CH and version.server/TXT/CH queries return any strings. If they do, the strings are logged and shown to the user.

However, if any of those strings contains bytes with the most significant bit set (i.e. in the range 0x80–0xFF inclusive), and if the backend is configured to store its results in a PostgreSQL database, the backend fails to return the test results to the GUI. The get_test_results RPCAPI method fails because of an internal server error. According to the version of Perl that is running of the system, one of the two error messages below are logged in the RPCAPI logs. Either:

[ERROR] [Zonemaster::Backend::RPCAPI] Caught Zonemaster::Backend::Error::Internal in the `Zonemaster::Backend::RPCAPI::get_test_results` method: Wide character at /usr/lib64/perl5/vendor_perl/Encode.pm line 296. Extra parameters: {'rpc_method' => 'get_test_results','reason' => 'Wide character at /usr/lib64/perl5/vendor_perl/Encode.pm line 296.
','error' => 'Zonemaster::Backend::Error::Internal','code' => -32603,'method' => 'Zonemaster::Backend::RPCAPI::get_test_results'}

Or:

[ERROR] [Zonemaster::Backend::RPCAPI] Caught Zonemaster::Backend::Error::Internal in the `Zonemaster::Backend::RPCAPI::get_test_results` method: Wide character at /home/vanderwal/.perl5/lib/perl5/Zonemaster/Backend/Translator.pm line 45. Extra parameters: {'rpc_method' => 'get_test_results','method' => 'Zonemaster::Backend::RPCAPI::get_test_results','error' => 'Zonemaster::Backend::Error::Internal','reason' => 'Wide character at /home/vanderwal/.perl5/lib/perl5/Zonemaster/Backend/Translator.pm line 45.
','code' => -32603}

The root cause of the problem is that the strings obtained by the Nameserver15 test case are improperly assumed to be character strings of an indeterminate encoding, instead of byte strings. When computing the translation of the N15_SOFTWARE_VERSION message, these strings are interpolated as is in a character string, without any form of prior sanitation. This step fails because the Zonemaster::Backend::Translator expects all strings to be character strings, not byte strings.

A contributing factor is presumably also the difference between PostgreSQL and SQLite: the former has a dedicated JSONB type which returns JSON objects as byte strings, whereas SQLite stores JSON as text, i.e. sequences of Unicode code points.

We should also consider using the DNS presentation format for reporting on such byte strings, by backslash-escaping non-ASCII bytes and backslash characters, before feeding them to the Translator module.

Installations using SQLite as backend database are unaffected (I haven’t tested with MySQL). Tests only using zonemaster-cli are unaffected.

Reproduction

In the wild

One domain triggering this problem is s42.re. When testing it on https://zonemaster.fr or https://zonemaster.net, the test completes but the GUI will not display the results. This is because the version.bind/TXT/CH string contains the byte sequence 0xef 0xbf 0xbd (the UTF-8 byte sequence for encoding U+FFFD � REPLACEMENT CHARACTER), which consists of non-ASCII bytes.

In the lab

The following script demonstrates the error:

#!/usr/bin/env perl
use v5.10;
binmode STDOUT, ':encoding(UTF-8)';

use Zonemaster::Backend::Translator;

my $string = pack('U', 0xFFFD);
my $message = {
        module => 'Nameserver',
        testcase => 'Nameserver15',
        timestamp => '15',
        level => 'NOTICE',
        tag => 'N15_SOFTWARE_VERSION',
        args => { ns_list => 'ns1.example/192.0.2.1', query_name => 'version.bind', 'string' => $string }
};

my $translator = Zonemaster::Backend::Translator->instance();
$translator->locale('en_US.UTF-8');
say $translator->translate_tag($message);

Running this script should display a message, but instead displays an error:

perl ./crash.pl
Wide character at /usr/lib64/perl5/vendor_perl/Encode.pm line 296.

Acknowledgements

I’d like to thank @mxsasha who found this issue and reported it privately to me for further investigation.