Skip to content
This repository has been archived by the owner on Apr 20, 2022. It is now read-only.

Latest commit

 

History

History
280 lines (191 loc) · 10.1 KB

TESTING.pod

File metadata and controls

280 lines (191 loc) · 10.1 KB
Raw

PHP Couchbase Testing

Documentation for test running and writing

Overview

The tests are written in a customized xunit-like framework, with an API compatible with a subset of PHPUnit.

They are designed and layed out in such a way to allow execution both under the phpt format (i.e. make test, run-tests.php) as well as phpunit.

Additionally, the tests included can be run from any version of the extension under both phpunit and phpt. This means that older versions and/or releases of the PHP extension may be tested using the latest version of the tests.

I'll try to outline various methods of running and/or writing the tests, as well as the test code layout itself.

Clarifications

Some mentions of "phpt tests" and "phpunit tests" will be made. These refer to the same exact tests; the difference being the test harness being used. phpt tests refer to the tests as they are run under run-tests.php whereas phpunit tests refer to the tests as they are run under phpunit.

Running

This will discuss running the actual tests

Common Test Setup (IMPORTANT)

The tests require a Couchbase server to connect to. By default these tests assume you have a server running on localhost with an unauthenticated default bucket.

If this is not the case, you need to tell the tests about your cluster setup. This can be done by create a tests/couchbase.local.inc file.

You can copy the template tests/couchbase.local.inc.dist and modify the defaults as appropriate.

PHPT Tests

Note: all command line examples assume the current directory is the source root of the php extension

The phpt tests can be run simply by doing:

$ make test

Which will run the entire test suite.

You can run a portion of the tests by using the TESTS Make variable, like so

# Run only the observe tests
$ make test TESTS=tests/phpt/Observe

Tests are largely organized under logical groups corresponding to the functionality they test (see "Framework").

Running From A Different Version

As shown, the TESTS Make variable lets you specify an alternate test location. This can be used to test from other versions; for example:

At the time of writing there are two branches of the PHP extension, one is 'master' and the other is '1.0.x'. The new shiny tests feature in the master branch but not in the '1.0.x' branch. It is still possible to run the current tests against the 1.0.x branch:

Assuming you are in a common top level directory containing ext-master ( which has the lastest 'master' branch) and ext-1.0.x (which has a version of the stable 1.0.x release) you can do like so:

cd ext-1.0.x
make tests TESTS=$PWD/../ext-master/tests

Helpful Information

The running of make test is rather cryptic. make test itself merely wraps another script run-tests.php (which is generated during the build process). While I haven't figured out how to run the script itself yet, the Makefile passes it options.

Passing arguments to run-tests via make test can be done through setting the environment variable TEST_PHP_ARGS.

For example, to run tests under valgrind, one may do

# This is going to take some time...
TEST_PHP_ARGS="-m" make test

To view test output of a failed test, simply navigate to the direct parent directory of the phpt file which was executed.

If the phpt test was tests/phpt/Connect/ConnectBasic.phpt then the test basename is ConnectBasic. You will then find a ConnectBasic.out ( actual output of the test), ConnectBasic.php (the script which was generated from the phpt file). ConnectBasic.sh (a shell script providing the correct incantation) and other such files providing information about the tests.

It might be helpful to check dmesg (on Linux) to see if a test is crashing :).

PHPUnit Tests

PHPUnit test invocation is a bit more involved but offers slightly more flexibility.

For any phpunit test to run, the environment variable EXTDIR must be set to the source tree directory for the extension, so that there will be a $EXTDIR/modules/couchbase.so after the module is built.

Additionally, due to the various complexities in injecting commandline options, the phpunit script itself is wrapped by a Perl script called runwrap.pl (this is located in the tests directory). This script passes all command line arguments verbatim to phpunit

The tests must be run from within the tests directory as well.

To run the entire suite one may do the following

export EXTDIR=$PWD
cd tests
./runwrap.pl -c test.xml
# whoopie!

Running Specific Tests

The tests are divided into classes (.inc files), and each test class contains multiple functions which comprise the specific test cases (these are actually generated into phpt files, see Framework).

To run a single test category, one only need pass the .inc file as an argument to the script; thus

./runwrap.pl Expiry.inc
# Will run all the Expiry tests

To run only a single test within the class:

./runwrap.pl --filter ExpiryTouchMulti Expiry.inc

Running Against Different Versions

To run the test suite against a different version, simply modify the EXTDIR environment variable to point to the source tree of the version being tested. The tests themselves must still be run from within the current tests directory though

Helpful Options and Information

By default phpunit does not create a new process for each test (unlike make test). This means that bugs which cause silent memory corruption will end up causing other tests to fail. Ususally this is what you want.

However if a specific test crashes it will cause the entire test suite to halt. phpunit accepts the --process-isolation option which will run each test in its own process.

Additionally, the runwrap.pl script understands a DEBUGGER environment variable which it will place before the php commandline. This is very useful for debugging and tracing the extension code:

# Try and reproduce a failing test, and debug it under gdb
$ DEBUGGER="gdb --args" ./runwrap.pl --filter ExpiryTouchMulti Expiry.inc
GNU gdb (GDB) 7.4.1-debian
# ....
(gdb) b lcb_connect
Function "lcb_connect" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (lcb_connect) pending.
(gdb) r
Starting program: /usr/bin/php /usr/bin/phpunit --filter ExpiryTouchMulti Expiry.inc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
PHPUnit 3.6.12 by Sebastian Bergmann.

Breakpoint 1, lcb_connect (instance=0xf8dcb0) at src/instance.c:1108
1108        release_socket(instance);
(gdb)

Or, run it under valgrind:

$ DEBUGGER="valgrind --leak-check=full" ./runwrap.pl --filter ExpiryTouchMulti Expiry.inc
==16110== Memcheck, a memory error detector
==16110== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==16110== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==16110== Command: php /usr/bin/phpunit --filter ExpiryTouchMulti Expiry.inc
==16110==
PHPUnit 3.6.12 by Sebastian Bergmann.

.

Time: 5 seconds, Memory: 3.00Mb

OK (1 test, 1 assertion)
==16110==
==16110== HEAP SUMMARY:
==16110==     in use at exit: 86,712 bytes in 2,582 blocks
==16110==   total heap usage: 40,660 allocs, 38,078 frees, 8,168,871 bytes allocated
==16110==
==16110== LEAK SUMMARY:
==16110==    definitely lost: 0 bytes in 0 blocks
==16110==    indirectly lost: 0 bytes in 0 blocks
==16110==      possibly lost: 0 bytes in 0 blocks
==16110==    still reachable: 86,712 bytes in 2,582 blocks
==16110==         suppressed: 0 bytes in 0 blocks
==16110== Reachable blocks (those to which a pointer was found) are not shown.
==16110== To see them, rerun with: --leak-check=full --show-reachable=yes
==16110==
==16110== For counts of detected and suppressed errors, rerun with: -v
==16110== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 36 from 10)

Framework

This section discusses the testing framework and tests themselves.

The test framework consists of several core support files:

cbtestframework/CBTestFramework.inc - xUnit/phpunit API

cbtestframework/cbtest-phpt-loader.inc - glue script to load tests under phpt

Common.inc - base class for all tests. Contains handy functions which
    extensions to xunit

gen-phpt.inc - script to generate phpt files from the test classes

TEST_CLASSES a line delimited list of test classes from which phpt tests
    should be generated

The tests themselves are written in xunit/phpunit style. They are generated into phpt style by using the gen-phpt.inc script. In a phpt context, the harness spawns a script which in turn loads the test class itself and only executes the requested test.

As per phpt, failures and success depend on expected output, so it's important that any tests themselves do not output anything when there is no error.

The framework itself will output the single line PHP_COUCHBASE_OK at the completion of each test (only in phpt context), and this is the output the phpt system will expect.

As per phpunit, tests will only be considered if they are public class members and their method names begin with test. Therefore make sure not to name any function to start with test unless it's actually a top level test.

phpunit also seems to have issues with some doxygen-style tags (specifically, any method documented with the @test tag will also be considered a test).

In short:

  • Don't output anything

    This will offend phpt

  • Name your test methods as starting with test

    Not doing so will offend phpunit

If you create a new test class, ensure that it is added to the TEST_CLASSES file, and that you run gen-phpt.inc thereafter to generate a test for it.

Additionally, do not use any test method from phpt which is not present in the CBTestFramework.inc file. These tests will break under phpt.