Concurrency in C# Cookbook (2014)
Chapter 6. Testing
Testing is an essential part of software quality. Unit testing advocates in particular have become common in the last few years; it seems that you read or hear about it everywhere. Some promote test-driven development, which is a style of coding that ensures you have comprehensive tests when the application is complete. The benefits of unit testing on code quality and overall time to completion are well known, and yet (at the time of writing) most developers do not actually write unit tests.
I encourage you to write at least some unit tests, and start with the code where you feel the least confidence. In my personal experience, unit tests have given me two main advantages:
1. Better understanding of the code. You know that part of the application that works but you have no idea how? It’s always kind of in the back of your mind when the really weird bug reports come in. Writing unit tests for the “hard” code is a great way to get a clear understanding of how it works. After writing unit tests describing its behavior, the code is no longer mysterious; you end up with a set of unit tests that describe its behavior as well as the dependencies that code has on the rest of the code.
2. Greater confidence to make changes. Sooner or later, you’ll get that feature request that requires you to change the “scary” code, and you’ll no longer be able to pretend it isn’t there (I know how that feels; I’ve been there!). It’s best to be proactive: write the unit tests for the scary code before the feature request comes in; do it now rather than later. Once your unit tests are complete, you’ll have an early warning system that will alert you immediately if your changes break existing behavior.
Both of these advantages apply to your own code just as much as others’ code. I’m sure there are other advantages, too. Does unit testing decrease the frequency of bugs? Most likely. Does unit testing reduce the overall time on a project? Possibly. But the advantages I describe above are defininte; I experience them every time I write unit tests.
So, that’s my sales pitch for unit testing.
This chapter contains recipes that are all about testing. A lot of developers (even ones who normally write unit tests) shy away from testing concurrent code because they assume it’s hard. However, as these recipes will show, unit testing concurrent code is not as difficult as they think. Modern features and libraries, such as async and Rx, have put a lot of thought into testing, and it shows. I encourage you to use these recipes to write unit tests, especially if you’re new to concurrency (i.e., the new concurrent code is “hard” and/or “scary”).
6.1. Unit Testing async Methods
Problem
You have an async method that you need to unit test.
Solution
Most modern unit test frameworks support async Task unit test methods, including MSTest, NUnit, and xUnit. MSTest began support for these tests with Visual Studio 2012. If you use another unit test framework, you may have to upgrade to the latest version.
Here is an example of an async MSTest unit test:
[TestMethod]
public async Task MyMethodAsync_ReturnsFalse()
{
var objectUnderTest = ...;
bool result = await objectUnderTest.MyMethodAsync();
Assert.IsFalse(result);
}
The unit test framework will notice that the return type of the method is Task and will intelligently wait for the task to complete before marking the test “successful” or “failed.”
If your unit test framework does not support async Task unit tests, then it will need some help to wait for the asynchronous operation under test. One option is to use Task.Wait and unwrap the AggregateException if there is an error. I prefer to use the AsyncContext type from theNito.AsyncEx NuGet package:
[TestMethod]
public void MyMethodAsync_ReturnsFalse()
{
AsyncContext.Run(async () =>
{
var objectUnderTest = ...;
bool result = await objectUnderTest.MyMethodAsync();
Assert.IsFalse(result);
});
}
AsyncContext.Run will wait until all asynchronous methods complete.
Discussion
Mocking asynchronous dependencies can be a bit awkward at first. It’s a good idea to at least test how your methods respond to synchronous success (mocking with Task.FromResult), synchronous errors (mocking with TaskCompletionSource<T>), and asynchronous success (mocking with Task.Yield and a return value).
When testing asynchronous code, deadlocks and race conditions may surface more often than when testing synchronous code. I find the per-test timeout setting useful; in Visual Studio, you can add a test settings file to your solution that allows you to set individual test timeouts. The default value is quite high; I usually have a per-test timeout setting of two seconds.
TIP
The AsyncContext type is in the Nito.AsyncEx NuGet package.
See Also
Recipe 6.2 covers unit testing asynchronous methods expected to fail.
6.2. Unit Testing async Methods Expected to Fail
Problem
You need to write a unit test that checks for a specific failure of an async Task method.
Solution
If you’re doing desktop or server development, MSTest does support failure testing via the regular ExpectedExceptionAttribute:
// Not a recommended solution; see below.
[TestMethod]
[ExpectedException(typeof(DivideByZeroException))]
public async Task Divide_WhenDenominatorIsZero_ThrowsDivideByZero()
{
await MyClass.DivideAsync(4, 0);
}
However, this solution is not the best solution. For one thing, Windows Store applications do not have ExpectedException available for their unit tests. Another more philosophical problem is that ExpectedException is actually a poor design. The exception it expects may be thrown byany of the methods called by your unit test method. A better design checks that a particular piece of code throws that exception, not the unit test as a whole.
Microsoft has moved in this direction by removing ExpectedException from Windows Store unit tests, replacing it with Assert.ThrowsException<TException>. You can use it like this:
[TestMethod]
public async Task Divide_WhenDenominatorIsZero_ThrowsDivideByZero()
{
await Assert.ThrowsException<DivideByZeroException>(async () =>
{
await MyClass.DivideAsync(4, 0);
});
}
WARNING
Do not forget to await the task returned by ThrowsException! This will propagate any assertion failures that it detects. If you forget the await and ignore the compiler warning, your unit test will always silently succeed regardless of your method’s behavior.
Unfortunately, Microsoft only added ThrowsException to Windows Store unit test projects, and (as of this writing) several other unit test frameworks do not include an equivalent async-compatible ThrowsException. If you find yourself in this boat, you can create your own:
/// <summary>
/// Ensures that an asynchronous delegate throws an exception.
/// </summary>
/// <typeparam name="TException">
/// The type of exception to expect.
/// </typeparam>
/// <param name="action">The asynchronous delegate to test.</param>
/// <param name="allowDerivedTypes">
/// Whether derived types should be accepted.
/// </param>
public static async Task ThrowsExceptionAsync<TException>(Func<Task> action,
bool allowDerivedTypes = true)
{
try
{
await action();
Assert.Fail("Delegate did not throw expected exception " +
typeof(TException).Name + ".");
}
catch (Exception ex)
{
if (allowDerivedTypes && !(ex is TException))
Assert.Fail("Delegate threw exception of type " + ex.GetType().Name +
", but " + typeof(TException).Name +
" or a derived type was expected.");
if (!allowDerivedTypes && ex.GetType() != typeof(TException))
Assert.Fail("Delegate threw exception of type " + ex.GetType().Name +
", but " + typeof(TException).Name + " was expected.");
}
}
You can use the method just like the Windows Store MSTest Assert.ThrowsException<TException> method. Don’t forget to await the return value!
Discussion
Testing error handling is just as important as testing the successful scenarios. Some would even say more important, since the successful scenario is the one that everyone tries before the software is released. If your application is going to behave strangely, it will be due to an unexpected error situation.
However, I encourage developers to move away from ExpectedException. It’s better to test for an exception thrown at a specific point rather than testing for an exception at any time during the test. Instead of ExpectedException, use ThrowsException (or its equivalent in your unit test framework), or use the ThrowsExceptionAsync implementation above.
See Also
Recipe 6.1 covers the basics of unit testing asynchronous methods.
6.3. Unit Testing async void Methods
Problem
You have an async void method that you need to unit test.
Solution
Stop.
You should do your dead-level best to avoid this problem rather than solve it. If it is possible to change your async void method to an async Task method, then do so.
If your method must be async void (e.g., to satisfy an interface method signature), then consider writing two methods: an async Task method that contains all the logic, and an async void wrapper that just calls the async Task method and awaits the result. The async void method satisfies the architecture requirements, while the async Task method (with all the logic) is testable.
If it’s completely impossible to change your method and you absolutely must unit test an async void method, then there is a way to do it. You can use the AsyncContext class from the Nito.AsyncEx library:
// Not a recommended solution; see above.
[TestMethod]
public void MyMethodAsync_DoesNotThrow()
{
AsyncContext.Run(() =>
{
var objectUnderTest = ...;
objectUnderTest.MyMethodAsync();
});
}
The AsyncContext type will wait until all asynchronous operations complete (including async void methods) and will propagate exceptions that they raise.
TIP
The AsyncContext type is in the Nito.AsyncEx NuGet package.
Discussion
One of the key guidelines in async code is to avoid async void. I strongly recommend you refactor your code instead of using AsyncContext for unit testing async void methods.
See Also
Recipe 6.1 covers unit testing async Task methods.
6.4. Unit Testing Dataflow Meshes
Problem
You have a dataflow mesh in your application, and you need to verify it works correctly.
Solution
Dataflow meshes are independent: they have a lifetime of their own and are asynchronous by nature. So, the most natural way to test them is with an asynchronous unit test. The following unit test verifies the custom dataflow block from Recipe 4.6:
[TestMethod]
public async Task MyCustomBlock_AddsOneToDataItems()
{
var myCustomBlock = CreateMyCustomBlock();
myCustomBlock.Post(3);
myCustomBlock.Post(13);
myCustomBlock.Complete();
Assert.AreEqual(4, myCustomBlock.Receive());
Assert.AreEqual(14, myCustomBlock.Receive());
await myCustomBlock.Completion;
}
Unit testing failures is not quite as straightforward, unfortunately. This is because exceptions in dataflow meshes are wrapped in another AggregateException each time they are propagated to the next block. The following example uses a helper method to ensure that an exception will discard data and propagate through the custom block:
[TestMethod]
public async Task MyCustomBlock_Fault_DiscardsDataAndFaults()
{
var myCustomBlock = CreateMyCustomBlock();
myCustomBlock.Post(3);
myCustomBlock.Post(13);
myCustomBlock.Fault(new InvalidOperationException());
try
{
await myCustomBlock.Completion;
}
catch (AggregateException ex)
{
AssertExceptionIs<InvalidOperationException>(
ex.Flatten().InnerException, false);
}
}
public static void AssertExceptionIs<TException>(Exception ex,
bool allowDerivedTypes = true)
{
if (allowDerivedTypes && !(ex is TException))
Assert.Fail("Exception is of type " + ex.GetType().Name + ", but "
+ typeof(TException).Name + " or a derived type was expected.");
if (!allowDerivedTypes && ex.GetType() != typeof(TException))
Assert.Fail("Exception is of type " + ex.GetType().Name + ", but "
+ typeof(TException).Name + " was expected.");
}
Discussion
Unit testing of dataflow meshes directly is doable, but somewhat awkward. If your mesh is a part of a larger component, then you may find that it’s easier to just unit test the larger component (implicitly testing the mesh). But if you’re developing a reusable custom block or mesh, then unit tests like the preceding ones should be used.
See Also
Recipe 6.1 covers unit testing async methods.
6.5. Unit Testing Rx Observables
Problem
Part of your program is using IObservable<T>, and you need to find a way to unit test it.
Solution
Reactive Extensions has a number of operators that produce sequences (e.g., Return) and other operators that can convert a reactive sequence into a regular collection or item (e.g., SingleAsync). We will use operators like Return to create stubs for observable dependencies, and operators like SingleAsync to test the output.
Consider the following code, which takes an HTTP service as a dependency and applies a timeout to the HTTP call:
public interface IHttpService
{
IObservable<string> GetString(string url);
}
public class MyTimeoutClass
{
private readonly IHttpService _httpService;
public MyTimeoutClass(IHttpService httpService)
{
_httpService = httpService;
}
public IObservable<string> GetStringWithTimeout(string url)
{
return _httpService.GetString(url)
.Timeout(TimeSpan.FromSeconds(1));
}
}
The code we wish to test is MyTimeoutClass, which consumes an observable dependency and produces an observable as output.
The Return operator creates a cold sequence with a single element in it; we can use this to build a simple stub. The SingleAsync operator returns a Task<T> that is completed when the next event arrives. SingleAsync can be used for simple unit tests like this:
class SuccessHttpServiceStub : IHttpService
{
public IObservable<string> GetString(string url)
{
return Observable.Return("stub");
}
}
[TestMethod]
public async Task MyTimeoutClass_SuccessfulGet_ReturnsResult()
{
var stub = new SuccessHttpServiceStub();
var my = new MyTimeoutClass(stub);
var result = await my.GetStringWithTimeout("http://www.example.com/")
.SingleAsync();
Assert.AreEqual("stub", result);
}
Another operator important in stub code is Throw, which returns an observable that ends with an error. This allows us to unit test the error case as well. The following example uses the ThrowsExceptionAsync helper from Recipe 6.2:
private class FailureHttpServiceStub : IHttpService
{
public IObservable<string> GetString(string url)
{
return Observable.Throw<string>(new HttpRequestException());
}
}
[TestMethod]
public async Task MyTimeoutClass_FailedGet_PropagatesFailure()
{
var stub = new FailureHttpServiceStub();
var my = new MyTimeoutClass(stub);
await ThrowsExceptionAsync<HttpRequestException>(async () =>
{
await my.GetStringWithTimeout("http://www.example.com/")
.SingleAsync();
});
}
Discussion
Return and Throw are great for creating observable stubs, and SingleAsync is an easy way to test observables with async unit tests. They’re a good combination for simple observables, but they don’t hold up well once you start dealing with time. For example, if we wanted to test the timeout capability of MyTimeoutClass, the unit test would actually have to wait for that amount of time. This doesn’t scale well as we add more unit tests. In Recipe 6.6, we’ll look at a special way that Reactive Extensions allow us to stub out time itself.
See Also
Recipe 6.1 covers unit testing async methods, which is very similar to unit tests that await SingleAsync.
Recipe 6.6 covers unit testing observable sequences that depend on time passing.
6.6. Unit Testing Rx Observables with Faked Scheduling
Problem
You have an observable that is dependent on time, and want to write a unit test that is not dependent on time. Observables that depend on time include ones that use timeouts, windowing/buffering, and throttling/sampling. You want to unit test these but do not want your unit tests to have excessive runtimes.
Solution
It’s certainly possible to put delays in your unit tests; however, there are two problems with that approach: 1) the unit tests take a long time to run, and 2) there are race conditions because the unit tests all run at the same time, making timing unpredictable.
The Rx library was designed with testing in mind; in fact, the Rx library itself is extensively unit tested. To enable this, Rx introduced a concept called a scheduler, and every Rx operator that deals with time is implemented using this abstract scheduler.
To make your observables testable, you need to allow your caller to specify the scheduler. For example, we can take the MyTimeoutClass from Recipe 6.5 and add a scheduler:
public interface IHttpService
{
IObservable<string> GetString(string url);
}
public class MyTimeoutClass
{
private readonly IHttpService _httpService;
public MyTimeoutClass(IHttpService httpService)
{
_httpService = httpService;
}
public IObservable<string> GetStringWithTimeout(string url,
IScheduler scheduler = null)
{
return _httpService.GetString(url)
.Timeout(TimeSpan.FromSeconds(1), scheduler ?? Scheduler.Default);
}
}
Next, let’s modify our HTTP service stub so that it also understands scheduling, and we’ll introduce a variable delay:
private class SuccessHttpServiceStub : IHttpService
{
public IScheduler Scheduler { get; set; }
public TimeSpan Delay { get; set; }
public IObservable<string> GetString(string url)
{
return Observable.Return("stub")
.Delay(Delay, Scheduler);
}
}
Now you can use TestScheduler, a type included in the Rx library. TestScheduler gives you powerful control over (virtual) time.
TIP
TestScheduler is in a separate NuGet package from the rest of Rx; you’ll need to install the Rx-Testing NuGet package.
TestScheduler gives you complete control over time, but you often just need to set up your code and then call TestScheduler.Start. Start will virtually advance time until everything is done. A simple success test case could look like this:
[TestMethod]
public void MyTimeoutClass_SuccessfulGetShortDelay_ReturnsResult()
{
var scheduler = new TestScheduler();
var stub = new SuccessHttpServiceStub
{
Scheduler = scheduler,
Delay = TimeSpan.FromSeconds(0.5),
};
var my = new MyTimeoutClass(stub);
string result = null;
my.GetStringWithTimeout("http://www.example.com/", scheduler)
.Subscribe(r => { result = r; });
scheduler.Start();
Assert.AreEqual("stub", result);
}
The code simulates a network delay of half a second. It’s important to note that this unit test does not take half a second to run; on my machine, it takes about 70 milliseconds. The half-second delay only exists in virtual time. The other notable difference in this unit test is that it is not asynchronous; since we are using TestScheduler, all our tests can complete immediately.
Now that everything is using test schedulers, it’s easy to test timeout situations:
[TestMethod]
public void MyTimeoutClass_SuccessfulGetLongDelay_ThrowsTimeoutException()
{
var scheduler = new TestScheduler();
var stub = new SuccessHttpServiceStub
{
Scheduler = scheduler,
Delay = TimeSpan.FromSeconds(1.5),
};
var my = new MyTimeoutClass(stub);
Exception result = null;
my.GetStringWithTimeout("http://www.example.com/", scheduler)
.Subscribe(_ => Assert.Fail("Received value"), ex => { result = ex; });
scheduler.Start();
Assert.IsInstanceOfType(result, typeof(TimeoutException));
}
Once again, this unit test does not take 1 second (or 1.5 seconds) to run; it executes immediately using virtual time.
Discussion
We’ve just scratched the surface on Reactive Extensions schedulers and virtual time. I recommend that you start unit testing when you start writing Rx code; as your code grows more complex, rest assured that the Rx testing is capable of handling it.
TestScheduler also has AdvanceTo and AdvanceBy methods, which allow you to gradually step through virtual time. There may be situations where these are useful, but you should strive to have your unit tests only test one thing. For example, when testing a timeout, you could write a single unit test that advanced the TestScheduler partially and ensured the timeout did not happen early and then advanced the TestScheduler past the timeout value and ensured the timeout did happen. However, I prefer to have independent unit tests as much as possible, for example, one unit test ensuring that the timeout did not happen early, and a different unit test ensuring that the timeout did happen later.
See Also
Recipe 6.5 covers the basics of unit testing observable sequences.