What is mutation testing? ​

Security is the most important aspect of smart contracts. In SmartPy we have set up a complete testing system because we believe that testing is the best way to understand how code works and to spot errors. How can we be sure that the tests cover all the important cases? With mutation testing.

Conveyor belt metaphor ​

Imagine a factory that produces toy robots. At the end of its conveyor belt we have added machines for quality assurance: the first machine checks if the robot has a head, the second one the legs... But one day a robot without arms passes all the quality tests. This means that a test machine for the arms was missing.

To be sure not to forget any test, the manager adds mutation testing. It's a special machine that is placed before the quality test. It takes a robot on the conveyor belt and modifies it: by removing a component, by exchanging its arms and legs... If the quality tests do not notice any problem, then this special machine will report that a quality assurance test is missing.

Concrete presentation ​

Mutation testing works in three steps:

1. We mutate a smart contract.
2. We observe how the modification affects the test results.
3. If the tests still pass: there is a problem.

Mutations done:

• We remove a command.
• We reverse a boolean.
• We remove an error message from fail (sp.failwith, .open_some...).
• We remove a branch from a | or a &.
• We add a sp.else: sp.failwith(sp.unit) after a sp.if without sp.else.

Each mutation is applied to each suitable sub-command and sub-expression of a contract.

For example: the mutation "remove a command" is applied to the command number 1 of the code, the 3 steps of mutation testing are applied and then it starts again by removing the command number 2 on the original contract. And this until all the commands are passed by this mutation. This whole process is done for each mutation.

Usage ​

Currently mutation testing is supported from the SmartPy CLI only.

To add a mutation test, we add a new test containing a test_scenario and a mutation_test. In this mutation_test we add the names of the tests that should be checked.

@sp.add_test(name="Mutation1")
def test():
    s = sp.test_scenario()
    with s.mutation_test() as mt:
        mt.add_scenario("<my test>", contract_id=0)
# Replace <my test> with the name of the classic test we've already written.
# We make as many `add_scenario` as there are tests.

By default, the system considers the first contract originated in the tested scenario. To test another contract you have to indicate a different contract_id.

Commented example ​

Let's imagine a contract that increments or decrements an integer.

# example.py
class MyContract(sp.Contract):
    def __init__(self):
        self.init(0)

    @sp.entrypoint
    def increment(self):
        self.data += 1

    @sp.entrypoint
    def decrement(self):
        self.data -= 1

@sp.add_test(name="Test1")
def test():
    c1 = MyContract()
    sc = sp.test_scenario()
    sc += c1
    c1.increment()
    sc.verify(c1.data == 1)

@sp.add_test(name="Mutation1")
def test():
    s = sp.test_scenario()
    with s.mutation_test() as mt:
        mt.add_scenario("Test1")

The attentive reader will have noticed that this test is not complete. Let's see if mutation testing notices it.

The command SmartPy.sh test example.py example tell us:

# Error displayed when running SmartPy test
[error] A mutated contract passed all tests.
 (example.py, line 26)

Mutation testing has found a gap in the tests for the entrypoint decrement. Let's see the rest of the error:

Mutated path: entrypoint "decrement" > .
Mutated contracted:
import smartpy as sp
class Contract(sp.Contract):
  def __init__(self):
    self.init_type(sp.TInt)

  @sp.entrypoint
  def decrement(self):
    pass

  @sp.entrypoint
  def increment(self):
    self.data += 1

Here we see the mutated contract that passes all the tests while being different from the expected contract.

We can see that the entrypoint "decrement" has been transformed into pass. So we need to add a check that fails on the mutated contract.

Fixing it ​

In order to address the gap uncovered by mutation testing let's modify the "Test1" test.

# example.py
@sp.add_test(name="Test1")
def test():
    c1 = MyContract()
    sc = sp.test_scenario()
    sc += c1
    c1.increment()
    sc.verify(c1.data == 1)
    c1.decrement()
    sc.verify(c1.data == 0)

We run SmartPy.sh test example.py example again.

No errors. This means that mutation testing hasn't uncovered any gaps in our test coverage.

Alternative: add a new test ​

Instead of modifying the "Test1" test, we could have added a second scenario.

# example.py
# We rename `"Test1"` into `"Increment"`
@sp.add_test(name="Increment")
def test():
    c1 = MyContract()
    sc = sp.test_scenario()
    sc += c1
    c1.increment()
    sc.verify(c1.data == 1)

# We add a scenario `"Decrement"`.
@sp.add_test(name="Decrement")
def test():
    c1 = MyContract()
    sc = sp.test_scenario()
    sc += c1
    c1.decrement()
    sc.verify(c1.data == -1)

# We add the two scenarios in the `mutation_test`.
@sp.add_test(name="Mutation1")
def test():
    s = sp.test_scenario()
    with s.mutation_test() as mt:
        mt.add_scenario("Increment")
        mt.add_scenario("Decrement")

We run SmartPy.sh test example.py example again.

No errors are raised. This means we have written all the necessary tests.