Mutation testing ​

Mutation testing is a means of identifying gaps in test coverage. It works by applying mutations to a contract, i.e. altering different parts of it one-by-one. For example, it may randomly omit a command from a contract and see how this affects tests.

A mutation test can be defined as follows:

@sp.add_test(name="Mutation1")
def test():
    s = sp.test_scenario()
    with s.mutation_test() as mt:
        mt.add_scenario("Test1", contract_id=0)
        mt.add_scenario("Test2", contract_id=0)

The contract_ids in all added scenarios must refer to the same contract (i.e. they come from the same contract class). If contract_id is omitted, the first contract in the scenario is mutated.

During mutation testing the contract will be modified uniformly across the added scenarios. A search is conducted for a modified version of the contract that passes all tests. If this search succeeds, the mutation test fails citing the modified contract as a witness.

Mutations currently done include the following:

• Remove each command.

• Replace each command with sp.failwith(sp.unit).

• Replace each boolean b by: True, False, not b

• Replace each integer n by: 0, 1, -1, -n, n+1, n-1

• An expression of the form x + y, is replaced by x and y. This is true for all binary operators of compatible type. The analogous thing is done for unary and ternary primitives.

Currently mutation testing is supported from the SmartPy CLI only.