diff --git a/README.md b/README.md index 984efea9..f7f45fdb 100644 --- a/README.md +++ b/README.md @@ -2,75 +2,227 @@ [![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run) -### Test Suite - -Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini` -file. `suite.ini` is basic ini-file, that consists of one section `default`, -and a number of fields: - -* `core` -* `description` - Test Suite description -* `script` - shebang file to start tarantool with -* disables: - * `disabled` - tests that must be skipped - * `release_disabled` - tests that must be skipped when Tarantool has been - builded with `Release` - * `valgrind_disabled` - tests that must be skipped when Valgrind is enabled -* `lua_libs` - paths for lua files, that should be copied into the folder, - where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua - lua/2.lua`) -* `long_run` - mark tests as long, enabled only with `--long` option (delimited - with the space, e.g. `long_run=t1.test.lua t2.test.lua`) -* `config` - test configuration file name - -Field `core` must be one of: - -* `tarantool` - Test-Suite for Functional Testing -* `app` - Another functional Test-Suite -* `unittest` - Unit-Testing Test Suite - -### Test - -Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have -skip condition file `*.skipcond`. On first run (without `.result`) `.result` -is generated from output. Each run, in the beggining, `.skipcond` file is -executed. In the local env there's object `self`, that's `Test` object. If test -must be skipped - you must put `self.skip = 1` in this file. Next, -`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is -compared with `.result`. If something differs, then 15 last string of this diff -file are printed and `.reject` file is saving in the `/rejects/` -subfolder given in options or set localy as `var/rejects/` by default. -If not, then `.reject` file is deleted. - -### Test configuration - -Test configuration file contains config for multiple run. For each test section -system runs separated test and compares result with common `.result` file. For -example we need to run one test for different db engines("*" means default -configuration): + + + +## Table of Contents + +- [Test suite configuration file](#test-suite-configuration-file) + - [General configuration values](#general-configuration-values) + - [Disabling and skipping tests](#disabling-and-skipping-tests) + - [Other parameters](#other-parameters) + - [Fragile tests](#fragile-tests) +- [Test composition](#test-composition) +- [Test execution](#test-execution) +- [Test configuration](#test-configuration) +- [Writing various types of tests](#writing-various-types-of-tests) + - [Python tests](#python-tests) + - [Lua](#lua) + - [SQL](#sql) +- [Interaction with the test environment](#interaction-with-the-test-environment) +- [pretest_clean()](#pretest_clean) +- [Tags](#tags) +- [Projects that use test-run](#projects-that-use-test-run) + + + +## Test suite configuration file + +Test suite is a bunch of tests located in a directory with a `suite.ini` +file. The `suite.ini` is a basic ini-file with one section `[default]`. + +```ini +[default] +core = luatest +description = Example test suite using luatest +... +``` + +Below is a list of configuration values (fields) in the `suite.ini` file. + +### General configuration values + +* `core` — major type of tests in this suite. + Should have one of the following values: + + * `tarantool` — a test that runs `tarantool` commands from a file, + and then compares output with a reference file. + Often called a "diff-test". + * `app` — a Lua script test. + Most of such tests produce + [TAP13 output](http://testanything.org/tap-version-13-specification.html) + using the built-in `tap` module. + Test-run validates such output without a reference file. + Some tests produce non-TAP13 output, which is compared to a reference file. + * `luatest` — test suite using the [luatest](https://github.com/tarantool/luatest/) library. + * `unittest` — an executable test file. + +* `description` + +* `script` — The file with Tarantool commands, used in `core = tarantool` tests. + It is used to start the default server using `tarantoolctl`. + The value should be a file in the same directory as `suite.ini`, like `box.lua` or `master.lua`. + This setting is mandatory for test suites with `core = tarantool` + and ignored in other types of tests. + +* `config` — name of a test configuration file, for example, `engine.cfg`. + For details, see the [test configuration](#test-configuration) section below. + +* `lua_libs` — paths to Lua files that should be copied to the directory, + where a server is started. For example: + + ```ini + lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua + ``` + +### Disabling and skipping tests + +A number of fields are used to disable certain tests: + +* `disabled` — list of tests that should be skipped under all conditions. + +* `release_disabled` — list of tests that should only run when Tarantool is + built in the debug mode (with `-DCMAKE_BUILD_TYPE=Debug` and not `=RelWithDebInfo`). + Tests that use error injections, which are only available in debug builds, + should always be `release_disabled`. + +* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled + +* `long_run` — mark tests as long. Such tests will run only with `--long` option. + + ```ini + long_run = t1.test.lua t2.test.lua + ``` + +Note that there's also an option to conditionally skip a test with [.skipcond](#test-composition) files. + +### Other parameters + +* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel + +* `use_unix_sockets` (optional, `False` by default) — use UNIX sockets for console +* `use_unix_sockets_iproto` (optional, `False` by default) — use UNIX sockets for IProto. + +* `show_reproduce_content` (optional, `True` by default) — when set to `True`, + show the contents of the reproduce file for each failed test. + (Implemented in [#113](https://github.com/tarantool/test-run/issues/113)) + + Reproduce files are not required for investigating results of + tests that run each case in a separate Tarantool instance, + which is the preferred way to run tests. + For such tests it makes sense to set `show_reproduce_content = False`. + +### Fragile tests + +Tests which fail sometimes due to external reasons can be marked as fragile (flaky). +Test-run will retry each test if it fails. +It's recommended to provide a list of related +[tarantool/tarantool](https://github.com/tarantool/tarantool) issues +next to each fragile test. + +```ini +fragile = { + "retries": 10, + "tests": { + "tarantoolctl.test.lua": { + "issues": [ "gh-5059", "gh-5346" ], + }, + {...}, + {...}, + } +} +``` + +## Test composition + +Each test consists of the following files: + +* Test file: `.test.lua`, `_test.lua`, `.test.py`, or `.test.sql`. +* Reference file (optional): `.result` . +* Skip condition file (optional): `.skipcond`. + +Reference file contains saved test output. +It is required for tests with non-TAP13 output. +Running `test-run.py` with `--update-result` option will update +the `.result` files with new test output. + +The optional skip condition file is a Python script. +It is used to skip a test on some conditions, typically on a certain OS. + +In the local Python environment of a test run there's a `self` object, +which is an instance of the [`Test` class](./lib/test.py). +Set `self.skip = 1` to skip this test. +For example, to skip `sometest` on OpenBSD, +add the following `sometest.skipcond` file: + +```python +import platform + +# Disabled on OpenBSD due to fail #XXXX. +if platform.system() == 'OpenBSD': + self.skip = 1 +``` + + +## Test execution + +Running a test begins with executing the `.skipcond` file. +If after execution `self.skip` equals `1`, test-run skips this test. + +Next, the test file is executed and the output is written to a `.reject` file. +If the output is TAP13-compatible, test-run validates it. +Otherwise, test-run compares it with the `.result` file. +If there's a difference between `.reject` and `.result`, the test fails and +the last 15 lines of diff are printed to output. + +Whenever a test fails, the `.reject` file is saved +and the path to this file is printed to output. + +## Test configuration + +Test configuration file contains configuration for multiple runs. +For each test section, system runs a separate test and compares the result to the common `.result` file. + +For example, `my.test.lua` will run with two different sets of parameters: ```json { "my.test.lua": { "first": {"a": 1, "b": 2}, "second": {"a": 1, "b": 3} + } +} +``` + +A common case is to run a test with different DB engines. +In the example below: + +* `first.test.lua` will run only on the memtx engine; +* `second.test.lua` will run as is, without parameterizing; +* all other tests in the suite (`*`) will be parameterized to run on both memtx and vinyl engines. + +```json +{ + "first.test.lua": { + "memtx": {"engine": "memtx"} }, + "second.test.lua": {}, "*": { "memtx": {"engine": "memtx"}, - "sophia": {"engine": "sophia"} + "vinyl": {"engine": "vinyl"} } } ``` -In test case we can get configuration from inspector: +In the test case we can get configuration from the inspector: ```lua engine = test_run:get_cfg('engine') -- first run engine is 'memtx' --- second run engine is 'sophia' +-- second run engine is 'vinyl' ``` -"engine" value has a special meaning for *.test.sql files: if it is "memtx" or +"engine" value has a special meaning for `*.test.sql` files: if it is "memtx" or "vinyl", then the corresponding default engine will be set before executing commands from a test file. An engine is set with the following commands: @@ -79,11 +231,13 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau pragma sql_default_engine='memtx|vinyl' ``` -If the first fails, then the second will be executed. When both fails, fail the test. +If the first fails, then the second will be executed. When both fail, the test fails. + +## Writing various types of tests -#### Python +### Python tests -Files: `.test.py`, `.result` and `.skipcond`(optionaly). +Files: `.test.py`, `.result` and `.skipcond` (optionally). Environment: @@ -166,14 +320,14 @@ box.info.lsn ... ``` -#### Lua +### Lua Files: `.test.lua`, `.result` and `.skipcond`(optionaly). Tests interact only with `AdminConnection`. Supports some preprocessor functions (eg `delimiter`) **Delimiter example:** -``` +```lua env = require('test_run') test_run = env.new() box.schema.space.create('temp') @@ -194,7 +348,7 @@ test **Delimiter result:** -``` +```yaml env = require('test_run') test_run = env.new() box.schema.space.create('temp') @@ -255,14 +409,14 @@ function echo(...) \ end ``` -#### SQL +### SQL *.test.sql files are just SQL statements written line-by-line. It is possible to mix SQL and Lua commands using `\set language lua` and `\set language sql` commands. -##### Interaction with the test environment +## Interaction with the test environment In lua test you can use `test_run` module to interact with the test environment. @@ -337,7 +491,7 @@ test_run:cmd('setopt delimiter ""'); join(test_run, 30) ``` -### pretest_clean() +## pretest_clean() Nothing will be done before a Python test and for `core = unittest` test suites. @@ -353,7 +507,7 @@ The following files will be removed: * `*.inprogress` * `[0-9]*/` -### Tags +## Tags Usage: @@ -409,7 +563,7 @@ Unsupported features: * Marking unit tests with tags. * Multiline comments (use singleline ones for now). -### Used By +## Projects that use test-run - [Tarantool](https://github.com/tarantool/tarantool) - in-memory database and application server - [memcached](https://github.com/tarantool/memcached) - Memcached protocol 'wrapper' for Tarantool