How to Test if a Class is Thread Safe

Documentation

Testing if a class is thread safe consists of calling all methods in parallel from multiple threads. To reduce the number of thread interleavings and to make the assertions as easy as possible we use a separate test for each method combination.

Example: A Counter

Let us look at an easy data structure to see how to write a test, a counter:

public class CounterWithDataRace {
    private int count = 0;
    public void increment() {
        count++;
    }
    public int getCount() {
        return count;
    }
}

A Test for All Combinations

To test this class we need to test all combination of state changing methods and all combinations of state changing and reading methods. So for our example we need to test increment called in parallel with itself and increment called in parallel with getCount:

@Test
public void parallelIncrement()  {
    try (AllInterleavings allInterleavings =
                 new AllInterleavings("withDataRace.parallelIncrement")) {
        while (allInterleavings.hasNext()) {
            CounterWithDataRace counter = new CounterWithDataRace();
            runParallel(counter::increment,
                        counter::increment);
            assertThat(counter.getCount(),is(2))  ;
        }
    }
}

@Test
public void parallelIncrementAndGet()  {
    try (AllInterleavings allInterleavings =
                 new AllInterleavings("withDataRace.parallelIncrementAndGet")) {
        while (allInterleavings.hasNext()) {
            CounterWithDataRace counter = new CounterWithDataRace();
            runParallel(counter::increment,
                        () -> assertThat(counter.getCount(),either(is(1)).or(is(2))) );
        }
    }
}

The while (allInterleavings.hasNext()) executes the test for each possible thread interleaving. And the runParallel executes each runnable in a separate test.

To test the state changing method increment we check the count after both threads have ended using assertThat(counter.getCount(),is(2)) To test the parallel execution of the reading method getCount and state changing method increment we need to assert that we either read the value before the increment or after the increment call. We do this by using the hamcrest either method: assertThat(counter.getCount(),either(is(1)).or(is(2)))

The first Bug: A Data Race

If we run the test using maven or gradle, VMLens reports a data race for both tests, since the field count is not correctly synchronized. VMLens generates a report in the build directory in the subfolder vmlens-report. Opening the file index.html we see the result of all tests. The name shown in the report is the name we have given in the constructor of AllInterleavings:

By clicking on the name of a test we see the thread interleaving which led to data race. So for example if we click on counterWithDataRace.parallelIncrement we see which thread interleaving led to the data race in the test counterWithDataRace.parallelIncrement.

The second Bug: A Race Condition

Now let us see what happens if we mark the field count as volatile to fix the data race:

public class CounterWithRaceCondition {
    private volatile int count = 0;
    // methods stay the same
}

Now the test withDataRace.parallelIncrementAndGet succeeds. But the test withDataRace.parallelIncrement fails with the error:

[ERROR]   CounterWithRaceConditionCT.parallelIncrement:20 
Expected: is <2>
     but: was <1>

To see wyh the test failed we can look at the trace:

The problem is that ++ is not atomic. So there are interleavings where the two threads first read the count variable and the write to the variable. Basically one thread reading a stale value.

The Solution: Atomic Methods

To make the class thread safe we need to make the methods atomic. Other parallel executed methods should either see the state before or after the atomic method call. This is what we tested in our test. We checked for all combinations of method calls if our methods are atomic. For example for the combination of increment and getCount parallel executed atomic methods can have the following two executions:

If those two methods are atomic we have either the combination thread A calls increment and then thread B calls getCount or the other way around thread B calls getCount and thread A calls increment. Therfore when we check the result of getCount it should be either 0 or 1.

And for the test of the parallel execution of two increment calls we check after both threads have ended that the the count is really two. The easiest way to make our two methods atomic is to use the synchronized modifier:

public class ThreadSafeCounter {
    private int count = 0;
    public synchronized void increment() {
        count++;
    }
    public synchronized int getCount() {
        return count;
    }
}

The test for this class now succeeds.

Summary

Testing if a class is thread safe consists of the following steps:

test each combination of reading and writing method
and each combination of writing methods with itself and with other writing methods
for each combination we test if the methods are atomic by
for reading method we check in the same thread if the value read is either the value before or after the writing method was executed
and for two writing method we check the state after both thread were joined

The source code is available at vmlens-tutorial-maven

Tutorial Java Doc