The Art of Unit Testing, Second Edition: with examples in C# (2013)
Appendix. Tools and frameworks
This book wouldn’t be complete without an overview of some tools and basic techniques you can use while writing unit tests. From database testing to UI testing and web testing, this appendix lists tools you should consider. Some are used for integration testing, and some allow unit testing. I’ll also mention some that I think are good for beginners.
The tools and techniques listed here are arranged in the following categories:
· Isolation frameworks
· Test frameworks
o Test runners
o Test APIs
· Test helpers
· DI and IoC containers
· Database testing
· Web testing
· UI testing
· Thread-related testing
· Acceptance testing
An updated version of the list of tools and techniques can be found on the book’s website: http://ArtOfUnitTesting.com.
A.1. Isolation frameworks
Mock or isolation frameworks are the bread and butter of advanced unit testing scenarios. There are many to choose from, and that’s a great thing:
· Rhino Mocks
· Typemock Isolator
· Moles/Microsoft Fakes
The previous edition of this book contained the following tools, which I’ve removed due to being out of date or relevance:
Here’s a short description of each framework.
Moq is an open source isolation framework and has an API that tries to be both simple to learn and easy to use. The API was one of the first to follow the arrange-act-assert style (as opposed to the record-and-replay model in older frameworks) and relies heavily on .NET 3.5 and 4 features, such as lambdas and extension methods. You need to feel comfortable with using lambdas, and the same goes for the rest of the frameworks in this list.
It’s quite simple to learn. My only beef with it is that the word mock is scattered all over the API, making this confusing. I would have liked, at least, to see a differentiation between creating stubs versus mocks, or using the word fake instead of both, to get rid of the confusion.
You can learn about Moq at http://code.google.com/p/moq/ and install it as a NuGet Package.
A.1.2. Rhino Mocks
Rhino Mocks is a widely used, open source framework for mocks and stubs. Although in the previous edition of this book I recommended using it, I no longer do. Its development has all but stopped, and there are better, more lightweight, simpler, and better-designed frameworks out there. If you have a choice, don’t use it. Ayende, the creator, mentioned in a Tweet that he isn’t really working on it anymore.
You can get Rhino Mocks at http://ayende.com/projects/rhino-mocks.aspx.
A.1.3. Typemock Isolator
Typemock Isolator is a commercial unconstrained (can fake anything, see chapter 6) isolation framework that tries to remove the terms mocks and stubs from its vocabulary in favor of a more simple and terse API.
Isolator differs from most of the other frameworks by allowing you to isolate components from their dependencies regardless of how the system is designed (although it supports all the features the other frameworks have). This makes it ideal for people who are getting into unit testing and want an incremental approach to learning about design and testability. Because it doesn’t force you to design for testability, you can learn to write tests correctly and then move on to learning better design, without having to mix the two. It’s also the most costly of the unconstrained bunch, which it makes up for in usability and features for legacy code.
Typemock Isolator has two flavors: a constrained basic edition that’s free and has all the limitations of using a constrained framework (no statics, only virtuals, and so on) and an unconstrained, paid option that has few limits on what it can fake.
Full disclosure: I worked at Typemock between 2008 and 2010.
You can get Typemock Isolator at http://www.typemock.com.
JustMock, from Telerik, is a relatively new isolation framework that’s a very obvious competitor to Typemock Isolator. The two frameworks’ APIs are so similar in design that it should be relatively easy to move between them for the basic stuff. Like Typemock, JustMock has two flavors: a constrained free edition and an unconstrained, paid option that has few limits on what it can fake.
There’s a little roughness with the APIs, and it currently, as far as I’ve been able to try, doesn’t support recursive fakes—the ability to make a fake that returns a fake object that returns a fake object, without needing to specify things explicitly. Grab it atwww.telerik.com/products/mocking.aspx.
A.1.5. Microsoft Fakes (Moles)
Microsoft Fakes is a project that started out in Microsoft research as the answer to the question “How can we fake the filesystem and other things like SharePoint without needing to purchase a company like Typemock?” What emerged was a framework called Moles. Moles later grew into Microsoft Fakes and is included in some versions of Visual Studio.
MS Fakes is another unconstrained isolation framework, with no API for verification that something was called. In essence, it provides utilities to create stubs. If you’d like to assert that some object was called, you could do it, but the test code would look like a mess.
Like the previous unconstrained frameworks, MS Fakes allows you to create two types of fake objects: you either generate unconstrained classes that inherit and override from code that is already testable, or you use shims. Shims are unconstrained, and stubs, the generated classes, are constrained. Confused? Yes, me too. One of the reasons I don’t recommend that anyone but brave souls with nothing to lose use MS Fakes is because of the horrible usability factor. They’re just confusing to use. Plus, the maintainability of the tests that use either shims or stubs is in question. The generated stubs need to be regenerated every time you change your code under test, followed by changing the tests, and code that uses shims is very long and hard to read and thus hard to maintain. MS Fakes might be free and included with Visual Studio, but it will cost you a lot of money down the line in developer hours, fixing and trying to understand your tests.
Another important point: using MS Fakes forces you to use MSTest as your test framework. If you want to use another one, you’re out of luck.
If you need an unconstrained framework for writing tests that will need to last more than a week or two, choose JustMock or Typemock Isolator.
Learn more about MS Fakes at http://msdn.microsoft.com/en-us/library/hh549175.aspx.
NSubstitute is an open source constrained isolation framework. Its API is very simple to learn and remember, and it has very good documentation. Also good: errors are very detailed. Along with FakeItEasy, it’s my first choice of a constrained framework for a new project.
Learn more about NSubstitute at http://nsubstitute.github.com/ and install it as a NuGet package.
FakeItEasy has not only a great name but also a very nice API. It’s my current favorite along with NSubstitute for constrained frameworks, but its documentation isn’t as good as NSub’s. My favorite thing about its API is that everything you’d want to accomplish starts with the character A, for example:
var foo = A.Fake<IFoo>();
A.CallTo(() => foo.Bar()).MustHaveHappened();
Learn more about FakeItEasy at https://github.com/FakeItEasy/FakeItEasy/wiki and install it as a NuGet package.
Foq was created as the result of a need by F# programmers to create fakes in a way that’s usable and readable in F#. It’s a constrained isolation framework, able to create fakes of abstract classes and interfaces. I’ve personally not used it because I’ve never worked with F#, but it seems to be the only reasonable solution in that space, for that purpose. Learn more about Foq at https://foq.codeplex.com/ and install it as a NuGet package.
Isolator++ was built by Typemock as an unconstrained isolation framework for C++. It can fake static methods, private methods, and more in legacy C++ code. It’s another commercial product, and it seems to be the only one in this space with those abilities. Learn more about it atwww.typemock.com/what-is-isolator-pp.
A.2. Test frameworks
Test frameworks are composed of two types of functionality:
· Test runners execute the tests you write, give results, and allow you to know what went wrong where.
· Test APIs include the attributes or classes you need to inherit and assertion APIs.
Let’s look at each in turn.
Visual Studio test runners:
· MS test runner built into Visual Studio
· Typemock Isolator
· ContinuousTests (Mighty Moose)
Test and assertion APIs:
A.2.1. Mighty Moose (a.k.a. ContinuousTests) continuous runner
A previously commercial tool turned free, Mighty Moose is dedicated to giving feedback on tests and coverage continuously, like NCrunch.
· It runs the tests in a background thread.
· Tests are automatically run as you change code and save and compile.
· It has a smart algorithm to tell which tests need to be run based on which code was changed.
Unfortunately, it looks like development has stopped on this tool. Learn more at http://continuoustests.com.
A.2.2. NCrunch continuous runner
NCrunch continuous runner is a commercial tool dedicated to giving feedback on tests and coverage continuously. While a relative newcomer, NCrunch has made its way into my heart (I purchased a license) because of several nice features:
· It runs the tests in a background thread.
· Tests are automatically run as you change code, without even needing to save it.
· Green/red coverage dots next to both tests and production code let you know if the current production line you’re working on is covered by any test and if that test is currently failing.
· It’s very configurable, to the point of annoyance. Just remember, when the wizard initially comes up on a simple project, just hit Esc to get the defaults of running all the tests.
Learn more at www.ncrunch.net/.
A.2.3. Typemock Isolator test runner
This test runner is part of a commercial isolation framework called Typemock Isolator.
This extension tries to run the tests and show coverage at the same time, on every compilation. It’s very much in beta state and has inconsistent behavior. Perhaps one day it will be more helpful, but these days I tend to turn it off and just use the isolation frameworks APIs.
Learn more at http://Typemock.com.
A.2.4. CodeRush test runner
This test runner part of a commercial tool called CodeRush is a well-known plug-in for Visual Studio.
Like ReSharper, there are some nice pros for this runner:
· It’s nicely integrated into the Visual Studio code editor by showing marks near a test that you can click to run individual tests.
· It supports most of the test APIs out there in .NET.
· If you’re already using CodeRush, it’s good enough.
Like ReSharper, the visual nature of the test results can hinder the experience for experienced TDD-ers. The tree of running tests, and showing all the results by default, even of passing tests, wastes time when you’re in the flow of TDD. But some people like it. Your mileage may vary.
Learn more at www.devexpress.com/Products/Visual_Studio_Add-in/Coding_Assistance/unit_test_runner.xml.
A.2.5. ReSharper test runner
This test runner is part of a commercial tool called ReSharper, a well-known plug-in for Visual Studio.
There are some nice pros for this runner:
· It’s nicely integrated into the Visual Studio code editor by showing marks near a test that you can click to run individual tests.
· It supports most of the test APIs out there in .NET.
· If you’re already using ReSharper, it’s good enough.
What I consider a con is the overly visual nature of the test results. The tree of running the tests is very nice and colorful. But painting it, and showing all the results by default, even of passing tests, wastes time when you’re in the flow of TDD. But some people like it. Your mileage may vary.
Learn more at www.jetbrains.com/resharper/features/unit_testing.html.
A.2.6. TestDriven.NET runner
This is a commercial test runner (free for personal use). This used to be my favorite test runner until I started using NCrunch. There’s a lot to love:
· It’s able to run tests for most, if not all, the test API frameworks out there in .NET, including NUnit, MSTest, xUnit.net, as well as some of the BDD API frameworks.
· It’s a little package. It’s a very small install and a minimalistic interface. The output is simple: it appears in the output window of Visual Studio, and some text is on the bottom sidebars of Visual Studio.
· It’s very fast, one of the fastest test runners.
· It has a unique ability: you can right-click any piece of code (not just tests) and select Test with > Debugger. You will then step into any code (even production code, even if it doesn’t have tests). Underneath TD.NET invoke the method you’re currently in using reflection, and it provides default values for the method if parameters are needed. This saves a lot of time in legacy code.
It is recommended to assign a shortcut to the TD.NET ReRunTests command in Visual Studio so that the flow of TDD is as smooth as possible.
A.2.7. NUnit GUI runner
The NUnit GUI runner is free and open source. This runner isn’t integrated into Visual Studio, so you have to run it from your desktop. Because of this, almost nobody uses it when they have the other options listed here integrated into Visual Studio. It’s crude and unpolished and not recommended.
A.2.8. MSTest runner
The MSTest runner comes built in with all versions of Visual Studio. In the paid versions it also has a plug-in mechanism that allows you to add support for running tests written in other test APIs such as NUnit or xUnit.net via special adapters that you can install as Visual Studio extensions.
One factor in favor of this runner is that it’s integrated into the Visual Studio Team System tool suite and provides good reporting, coverage, and build automation out of the box. If your company uses Team System for automated builds, try MSTest as your test runner in the nightly and CI builds because of the good integration possibilities such as reporting.
Two areas where MSTest lacks are performance and dependencies:
· Dependencies —To run your tests using mstest.exe, you need to have Visual Studio installed on your build machine. That might be OK for some, especially if where you compile is the same place you run your tests. But if you want to run your tests in a relatively clean environment, in already compiled form, this can be overkill and problematic if you want your tests to run in an environment that specifically does not have Visual Studio installed.
· Slow —The tests in MSTest do a lot under the hood before and after each test, copying files, running external processes, profiling, and more, and this makes MSTest feel like the slowest runner of all the ones I’ve ever used.
Pex (short for program exploration) is an intelligent assistant to the programmer. From a parameterized unit test, it automatically produces a traditional unit test suite with high code coverage. In addition, it suggests to the programmer how to fix the bugs.
With Pex, you can create special tests that have parameters in them and put special attributes on those tests. The Pex engine will generate new tests that you can later run as part of your test suite. It’s great for finding corner cases and edge conditions that aren’t handled properly in your code. You should use Pex in addition to a regular test framework, such as NUnit or MbUnit.
You can get Pex at http://research.microsoft.com/projects/pex/.
A.3. Test APIs
The next batch of tools provides higher-level abstractions and wrappers for the base unit testing frameworks.
A.3.1. MSTest API—Microsoft’s unit testing framework
This comes bundled with any version of Visual Studio .NET Professional or above. It includes basic features that are similar to NUnit.
But several problems make MSTest an inferior product for unit testing compared to NUnit or xUnit.net:
· Lack of Assert.Throws
One big problem with this framework is that it’s not as easily extensible as the other testing frameworks. Although there have been several online discussions in the past about making MSTest more extensible with custom test attributes, it seems that the Visual Studio team has all but given up on making MSTest a viable alternative to NUnit and others.
Instead, VS 2012 features a plug-in mechanism that allows you to use NUnit or any other test framework as your default test framework, with the MSTest runner running your NUnit tests. There are already adapters available for NUnit and xUnit.net (NUnit test adapter or xUnit.net runner for Visual Studio 2012) if you just want to use the MSTest runner with other frameworks. Unfortunately, the express, free version of Visual Studio doesn’t contain this mechanism, forcing you to use the inferior MSTest. (On a side note, why does Microsoft force you to purchase Visual Studio so that you can develop code that makes the MS platform more dominant?)
Lack of Assert.Throws
This is a simple matter. In MSTest you have an ExpectedException attribute, but you don’t have Assert.Throws, which allows testing that a specific line threw an exception. Over six years after inception, and four years after most other frameworks, this framework’s developers haven’t bothered adding this literally 10 lines of code implementation to it, leaving me wondering how much they really care about unit tests.
A.3.2. MSTest for Metro Apps (Windows Store)
MSTest for Metro Apps is an API for writing Windows Store apps that looks like MSTest but seems to get the right idea regarding unit tests. For example, it sports its own version of Asset.ThrowsException().
It seems like you’re forced to use this framework for writing Windows Store apps with unit tests, but a solution exists if you use linked projects. For information see http://stackoverflow.com/questions/12924579/testing-a-windows-8-store-app-with-nunit.
A.3.3. NUnit API
NUnit is currently the de facto test API framework for unit test developers in .NET. It’s open source and is in almost ubiquitous use among those who do unit testing. I cover NUnit in depth in chapter 2. NUnit is easily extensible and has a large user base and forums. I’d recommend it to anyone starting out with unit testing in .NET. I still use it today.
You can get NUnit at www.Nunit.org.
xUnit.net is an open source test API framework, developed in cooperation with one of the original authors of NUnit, Jim Newkirk. It’s a minimalist and elegant test framework that tries to get back to basics by having fewer features, not more, than the other frameworks and by supporting different names on its attributes.
What’s so radically different about it? It has no setup or teardown methods, for one. You have to use the constructor and a dispose method on the test class. Another big difference is in how easy it is to extend.
Because xUnit.net reads so differently from the other frameworks, it takes a while to get used to if you’re coming from a framework like NUnit or MbUnit. If you’ve never used any test framework before, xUnit.net is easy to grasp and use, and it’s robust enough to be used in a real project.
For more information and download see www.codeplex.com/xunit.
A.3.5. Fluent Assertions helper API
The Fluent Assertions helper API is a new breed of test API. It’s a cute library that’s designed solely for one purpose: to allow you to assert on anything, regardless of the test API you’re using. For example, you can use it to get Assert.Throws()-like functionality in MSTest.
More information is available at http://fluentassertions.codeplex.com/.
A.3.6. Shouldly helper API
The Shouldly helper API is a lot like Fluent Assertions but smaller. It’s also designed solely for one purpose: to allow you to assert on anything, regardless of the test API you’re using. More information can be found at http://shouldly.github.com.
A.3.7. SharpTestsEx helper API
Like Fluent Assertions, the SharpTestsEx helper API is designed solely for one purpose: to allow you to assert on anything, regardless of the test API you’re using. More information is available at http://sharptestex.codeplex.com.
A.3.8. AutoFixture helper API
The AutoFixture helper API is not an assertion API. AutoFixture is designed to make it easier to create objects under test that you don’t care about. For example, you need some number or some string. Think of it as a smart factory that can inject objects and input values into your test.
I’ve looked at using it, and the thing I find most appealing about it is the ability to create an instance of the class under test without knowing what its constructor signature looks like, which can make my test more maintainable over time. Still, that’s not enough reason for me to use it, because I can simply do that with a small factory method in my tests.
Also, it scares me a bit to let it inject random values into my tests, because it makes me run a different test each time I run it. It also complicates my asserts, because then I have to calculate that my expected output must be based on the random injected parameters, which may lead to repeating production code logic in my tests.
More information can be found at https://github.com/AutoFixture/AutoFixture.
A.4. IoC containers
IoC containers can be used to improve the architectural qualities of an object-oriented system by reducing the mechanical costs of good design techniques (such as using constructor parameters, managing object lifetimes, and so on).
Containers can enable looser coupling between classes and their dependencies, improve the testability of a class structure, and provide generic flexibility mechanisms. Used judiciously, containers can greatly enhance the opportunities for code reuse by minimizing direct coupling between classes and configuration mechanisms (such as by using interfaces).
There are a lot of those in the .NET space. And they’re varied and interesting to take a look at. Performance-wise, if you care much about that, there’s a good comparison of them all at http://www.palmmedia.de/Blog/2011/8/30/ioc-container-benchmark-performance-comparison. Personally, I never felt that IoC containers were my root cause of performance issues, and if that were ever the case, that would be a very good place to be.
Anyway, there are lots of them, but we’ll look at the following tools, which are used frequently in the community.
I chose the tools to cover based on a usage poll I took on my blog during March 2013. Here’s the top of the results heap for usage:
· Autofac (Auto Factory)
· Castle Windsor
· Microsoft Unity
· Microsoft Managed Extensibility Framework
Let’s look briefly at each of these frameworks.
Autofac was one of the first to offer a fresh approach to IoC in .NET that fits well with the C# 3 and 4 syntax. It takes a rather minimalistic approach in terms of APIs. The API is radically different from those of the other frameworks and requires a bit of getting used to. It also requires .NET 3.5 to work, and you’ll need a good knowledge of lambda syntax. Autofac is difficult to explain, so you’ll have to go to the site to see how different it is. I recommend it for people who already have experience with other DI frameworks.
You can get it at http://code.google.com/p/autofac/.
Ninject also has simple syntax and good usability. There isn’t much else to say about it except that I highly recommend taking a look at it.
You can find out more about Ninject at http://ninject.org/.
A.4.3. Castle Windsor
Castle is a large, open source project that covers a lot of areas. Windsor is one of those areas, and it provides a mature and powerful implementation of a DI container.
Castle Windsor contains most of the features you’ll ever want in a container and more, but it has a relatively high learning curve because of all the features.
You can learn about the Castle Windsor container at http://docs.castleproject.org/Windsor.MainPage.ashx.
A.4.4. Microsoft Unity
Unity is a latecomer to the DI container field, but it provides a simple and minimal approach that can be easily learned and used by beginners. Advanced users may find it lacking, but it certainly answers my 80–20 rule: it provides 80% of the features you look for most of the time.
Unity is open source by Microsoft, and it has good documentation. I’d recommend it as a starting point for working with containers.
You can get Unity at www.codeplex.com/unity.
StructureMap is an open source container framework written by Jeremy D. Miller. Its API is very fluent and tries to mimic natural language and generic constructs as much as possible.
The current documentation on it is lacking, but it contains powerful features, such as a built-in automocking container (a container that can create stubs automatically when requested to by the test), powerful lifetime management, XML-free configuration, integration with ASP.NET, and more.
You can get StructureMap at http://structuremap.net.
A.4.6. Microsoft Managed Extensibility Framework
The Managed Extensibility Framework (MEF) isn’t actually a container, but it does fall in the same general category of providing services that instantiate classes in your code. It’s designed to be much more than a container; it’s a full plug-in model for small and large applications. MEF includes a lightweight IoC container framework so you can easily inject dependencies into various places in your code by using special attributes.
MEF does involve a bit of a learning curve, and I wouldn’t recommend using it strictly as an IoC container. If you do use it for extensibility features in your application, it can also be used as a DI container.
You can get MEF at http://mef.codeplex.com/.
A.5. Database testing
How to do database testing is a burning question for those who are starting out. Many questions arise such as, “Should I stub out the database in my tests?” This section provides some guidelines.
First, let’s talk about doing integration tests against the database.
A.5.1. Use integration tests for your data layer
How should you test your data layer? Should you abstract away the database interfaces? Should you use the real database?
I usually write integration-style tests for the data layer (the part of the app structure that talks directly to the database) in my applications because data logic is almost always divided between the application logic and the database itself (triggers, security rules, referential integrity, and so on). Unless you can test the database logic in complete isolation (and I’ve found no really good framework for this purpose), the only way to make sure it works in tests is to couple testing the data-layer logic to the real database.
Testing the data layer and the database together leaves few surprises for later in the project. But testing against the database has its problems, the main one being that you’re testing against state shared by many tests. If you insert a line into the database in one test, the next test can see that line as well.
What you need is a way to roll back the changes you make to the database, and thankfully, there’s an easy way to do it in the .NET Framework.
A.5.2. Use TransactionScope to roll back changes to data
The TransactionScope class is smart enough to handle very complicated transactions, as well as nested transactions where your code under test calls commits on its own local transaction.
Here’s a simple piece of code that shows how easy it is to add rollback ability to your tests:
public class TrannsactionScopeTests
private TransactionScope trans = null;
public void SetUp()
trans = new TransactionScope(TransactionScopeOption.Required);
public void TearDown()
public void TestServicedSameTransaction()
MySimpleClass c = new MySimpleClass();
long id = c.InsertCategoryStandard("whatever");
long id2 = c.InsertCategoryStandard("whatever");
Console.WriteLine("Got id of " + id);
Console.WriteLine("Got id of " + id2);
You set up a transaction scope in the setup and dispose of it in the teardown.
By not committing it at the test class level, you basically roll back any changes to the database, because dispose initiates a database rollback if commit wasn’t called first.
Some feel that another good option is to run the tests against an in-memory database. My feelings on that are mixed. On the one hand, it’s closer to reality, in that you also test the database logic. On the other hand, if your application uses a different database engine, with different features, there’s a big chance that some things will pass or fail during tests with the in-memory database and will work differently in production. I choose to work with whatever is as close to the real thing as possible. Usually that means using the same database engine.
If the in-memory database engine has the same features and logic embedded in it, it might be a great idea.
A.6. Web testing
“How do I test my web pages?” is another question that comes up a lot. Here are some tools that can help you in this quest:
· Team System Web Test
Following is a short description of each tool.
Ivonna is a unit testing-helping framework that abstracts away the need to run ASP.NET-related tests using a real HTTP session and pages. It does some powerful things behind the scenes, such as compiling pages that you want to test and letting you test controls inside them without needing a browser session, and it fakes the full HTTP runtime model.
You write the code in your unit tests just like you’re testing other in-memory objects. There’s no need for a web server and such nonsense.
Ivonna is being developed in partnership with Typemock and runs as an add-on to the Typemock Isolator framework. You can get Ivonna at http://ivonna.biz.
A.6.2. Team System web test
Visual Studio Team Test and Team Suite editions include the powerful ability to record and replay web requests for pages and verify various things during these runs. This is strictly integration testing, but it’s really powerful. The latest versions also support recording Ajax actions on the page and make things much easier to test in terms of usability.
You can find more info on Team System at http://msdn.microsoft.com/en-us/teamsystem/default.aspx.
Watir (pronounced “water”) stands for “web application testing in Ruby.” It’s open source, and it allows scripting of browser actions using the Ruby programming language. Many Rubyists swear by it, but it does require that you learn a whole new language to use. A lot of .NET projects are using it successfully, so it’s not a big deal.
You can get Watir at http://watir.com/.
A.6.4. Selenium WebDriver
Selenium is a suite of tools designed to automate web app testing across many platforms. It’s older than all the other frameworks in this list, and it also has an API wrapper for .NET. WebDriver is an extension of it that fits many different kinds of browsers, including mobile ones. It’s very powerful.
Selenium is an integration testing framework, and it’s in wide use. It’s a good place to start. But beware: it has many features and the learning curve is high.
You can get it at http://docs.seleniumhq.org/projects/webdriver/.
Coypu is a .NET abstract on top of Selenium and other web-related testing tools. It’s quite new at the time of writing but might have plenty of potential. It might be worth checking it out.
Learn more at https://github.com/featurist/coypu.
Capybara is a Ruby-based tool that automates the browser. It allows you to use the RSpec (BDD-style) API to automate the browser, which many people find just lovely to read.
Selenium is more mature, but Capybara is more inviting and progressing quickly. When I do Ruby stuff, this is what I use.
Learn more at https://github.com/jnicklas/capybara.
Here’s a partial list of frameworks to look at:
· Jasmin —A very well-known BDD-style framework that I have used. I recommend it. http://pivotal.github.io/jasmine/
· Sinon.JS —Create fakes in JS. http://sinonjs.org/
· Mocha —Also very well known and used in many projects. http://visionmedia.github.com/mocha/
· QUnit —A bit long in the tooth but still a good test framework. http://qunitjs.com/
· Buster.JS —A very new framework. http://docs.busterjs.org/en/latest/
· Vows.js —An up and coming framework. https://github.com/cloudhead/vows
A.7. UI testing (desktop)
UI testing is always a difficult task. I’m not a great believer in writing unit tests or integration tests for UIs because the return is low compared to the amount of time you invest in writing them. UIs change too much to be testable in a consistent manner, as far as I’m concerned. That’s why I usually try to separate all the logic from the UI into a lower layer that I can test separately with standard unit testing techniques.
There are no tools I can heartily recommend (that won’t make you break the keyboard after three months) to look at in this space.
A.8. Thread-related testing
Threads have always been the bane of unit testing. They’re simply untestable. That’s why new frameworks are emerging that let you test thread-related logic (deadlocks, race conditions, and so on), such as these:
· Microsoft CHESS
I’ll give a brief rundown of each tool.
A.8.1. Microsoft CHESS
CHESS was an upcoming tool that’s now somewhat open source by Microsoft on Codeplex.com. CHESS attempts to find thread-related problems (deadlocks, hangs, livelocks, and more) in your code by running all relevant permutations of threads on existing code. These tests are written as simple unit tests.
Check it out at http://chesstool.codeplex.com.
This is a little open source framework I developed a while back. It allows you to run multiple threads during one test to see if anything weird happens to your code (deadlocks, for example). It isn’t feature complete, but it’s a good attempt at a multithreaded test (rather than a test for multithreaded code).
You can get it from my blog, at http://osherove.com/blog/2007/6/22/multi-threaded-unit-tests-with-osherovethreadtester.html.
A.9. Acceptance testing
Acceptance tests enhance collaboration between customers and developers in software development. They enable customers, testers, and programmers to learn what the software should do, and they automatically compare that to what it actually does. They compare customers’ expectations to actual results. It’s a great way to collaborate on complicated problems (and get them right) early in development.
Unfortunately, there are few frameworks for automated acceptance testing and just one that works these days! I’m hoping this will change soon. Here are the tools we’ll look at:
Let’s take a closer look.
FitNesse is a lightweight, open source framework that supposedly makes it easy for software teams to define acceptance tests—web pages containing simple tables of inputs and expected outputs—and to run those tests and see the results.
FitNesse is quite buggy, but it has been in use in many places with varying degrees of success. I personally haven’t gotten it working quite perfectly.
You can learn more about FitNesse at www.fitnesse.org.
SpecFlow tries to give the .NET world what Cucumber has given the Ruby world: a tool that allows you to write the specification language as simple text files, which you can then collaborate on with your customers and QA departments.
It does a pretty good job at that. Learn more at http://www.specflow.org.
Cucumber is a Ruby-based tool that allows you to write your specifications in a special language called Gherkin (yes, I agree). These are simple text files, and you then have to write special connector code to run actual code that acts on your application code.
It sounds complicated, but it isn’t.
So what’s it doing here if it’s a Ruby tool? It’s here because it has inspired a whole suite of tools in the .NET world, of which only one seems to be surviving right now—SpecFlow.
But there is a way to run Cucumber on .NET if you use IronRuby—a language abandoned by Microsoft and thrown to the open source world over the wall, never to be heard from again. (Great job!)
In any case, Cucumber is important enough to be aware of regardless of whether you intend to use it. It will help you understand why some things in .NET try to do the same thing.
Also, it’s the basis of the Gherkin language, which other tools will try to implement now and in the future. Learn more at http://cukes.info/.
TickSpec is for you if you use F#. I haven’t used it myself, because I haven’t used F#, but it’s meant as a similar framework relating to acceptance and BDD-styled frameworks, as mentioned previously. I’ve also not heard of others using it, but that may just be because I’m not that much in F# circles. Learn more at https://tickspec.codeplex.com/.
A.10. BDD-style API frameworks
The last few years have also given rise to a bunch of frameworks that imitate another tool from the Ruby world, called RSpec. This tool introduced the idea that maybe unit testing isn’t a great naming convention, and by changing it to BDD we can make things more readable and perhaps even converse more with our customers about it.
To my mind, the idea of implementing these frameworks simply as different APIs in which you’d write unit or integration tests already negates most of the possibility of conversing more with your customers about them (than before), because they’re not likely to really read your code or change it. I feel that the acceptance frameworks from the previous section fit more into that state of mind.
So this leaves us with just coders trying to use these APIs.
Because these APIs draw inspiration from the BDD-style language of Cucumber, in some cases they seem more readable, but to my mind, not the simple cases, which benefit more from simple assert-style tests. Your mileage may vary.
Here are some of the better-known BDD-style frameworks. I’m not creating a subsection of any of them, because I haven’t personally used any of them on a real project over a long period of time:
· NSpec is the oldest and seems in pretty good shape. Learn it at http://nspec.org/.
· StoryQ is another oldie but goodie. It produces very readable output and also has a tool that translated Gherkin stories to compliable test code. Learn it at http://storyq.codeplex.com/.
· MSpec, or Machine.Specifications, tries to be as close to the source (RSpec) as possible with many lambda tricks. It grows on you. Learn it at https://github.com/machine/machine.specifications.
· TickSpec is the same idea implemented for F#. Learn it at http://tickspec.codeplex.com/.