Codewars Python Test Framework V2

Before You Read This Page

This page documents newly proposed version of Python Test Framework (a.k.a. cw-2.py) which is not deployed to Codewars yet. If you want to create a Kata right now, please refer to the current Python Test Framework instead.

---

Basic Setup

@test.describe('Example Tests')
def example_tests():
    @test.it('Example Test Case')
    def example_test_case():
        test.assert_equals(add(1, 1), 2, 'Optional Message on Failure')

Available Assertions

Equality Test

test.assert_equals(actual, expected) # default message: <actual> should equal <expected>
test.assert_equals(actual, expected, message)

Checks that the actual value equals the expected value. This function is usually the main building block of a Kata's test cases.
Note that, since Python's equality operator checks for deep equality by default, you don't have to compare the individual contents yourself when you want to compare with a list, tuple, set, etc.

Non-equality Test

test.assert_not_equals(actual, unexpected) # default message: <actual> should not equal <expected>
test.assert_not_equals(actual, unexpected, message)

Checks that the actual value does not equal the (un)expected value.

Approximate Equality Test

test.assert_approx_equals(actual, expected, margin=1e-9, message=None)
# default message: <actual> should be close to <expected> with absolute or relative margin of <margin>

Checks if the actual value is sufficiently close to the expected. If the expected value is a float, some amount of floating-point error is always expected during the calculation. In this case, you're strongly advised to use this function to check the value instead of assert_equals.

Pass and Fail

test.pass_()
test.fail(message)

Simply generate a passed test, or a failed test with a message. If your test method is very complicated or you need a special procedure to test something, these functions are probably a good choice.

Truthness Test

test.expect(boolean) # default message: Value is not what was expected
test.expect(boolean, message)

Checks if the passed value is truthy. This function can be helpful when you test something which cannot be tested using other functions. However, since this function's default failure message is not helpful at all, you're strongly advised to provide your own helpful message.

Error Test

test.expect_error(message, function)

Checks that function throws something. It does not matter which exception the function exactly throws.

Utilities

Timeout Utility

@test.timeout(sec)
def some_function():
    do_some_computation()
    test_something_with_it()

Runs the function within the time limit. Generates a failed assertion when the function fails to complete in time, and its execution is terminated immediately. The time limit (in seconds) can be a float.

Test Suite Best Practices

Example Test Fixture

@describe('Fixed Tests')
def fixed_tests():

    # Basic Tests: Test the basic behavior (basic understanding of the task).
    @it('Basic Test Cases')
    def basic_tests():
        test.assert_equals(two_oldest_ages([2, 4, 6, 9, 12, 14]), [12, 14])

    # Edge Cases: Test the edge cases, which are not common but hard to correctly solve.
    # These are needed because "rare but hard cases" are not well-covered by random tests only.
    @it('Edge Cases')
    def edge_case_tests():
        test.assert_equals(two_oldest_ages([0, 0, 0, 0, 0, 0]), [0, 0])

# Random tests: Test the behavior against your reference solution.
# This is mainly to prevent the warrior passing the tests by hardcoding the fixed cases.
# The functions `random`, `randint`, `choice`, `shuffle`, and `sample` inside Python's `random` module will be helpful.
@describe('Random Tests')
def random_tests():

    # The reference solution should be placed here, in order to prevent the warrior from abusing your reference solution
    def _reference(ages):
        ...

    # When running a random test, you need to make sure that the expected value is computed first.
    # If the warrior's solution is run first, it may mutate the input list and thus easily bypass them.
    # Also, take extra care on your own reference solution to not mutate the input :)
    def _do_one_test():
        ages = generate_random_case()
        expected = _reference(ages)
        test.assert_equals(two_oldest_ages(ages), expected)
        # test.assert_equals(two_oldest_ages(ages), _reference(ages))

    # The number of random tests must be enough to test every possible aspects of the input.
    # The rule of thumb is 100 tests, but you have to think carefully according to the requirements of your Kata.
    @it('Random Test Cases')
    def random_test_cases():
        for _ in range(100):
            _do_one_test()