Programming Ruby 1.9 & 2.0: The Pragmatic Programmers’ Guide (2013)

Part 1. Facets of Ruby

Chapter 13. Unit Testing

Unit testing is testing that focuses on small chunks (units) of code, typically individual methods or lines within methods. This is in contrast to most other forms of testing, which consider the system as a whole.

Why focus in so tightly? It’s because ultimately all software is constructed in layers; code in one layer relies on the correct operation of the code in the layers below. If this underlying code turns out to contain bugs, then all higher layers are potentially affected. This is a big problem. Fred may write some code with a bug one week, and then you may end up calling it, indirectly, two months later. When your code generates incorrect results, it will take you a while to track down the problem in Fred’s method. And when you ask Fred why he wrote it that way, the likely answer will be “I don’t remember. That was months ago.”

If instead Fred had unit tested his code when he wrote it, two things would have happened. First, he’d have found the bug while the code was still fresh in his mind. Second, because the unit test was only looking at the code he’d just written, when the bug did appear, he’d only have to look through a handful of lines of code to find it, rather than doing archaeology on the rest of the code base.

Unit testing helps developers write better code. It helps before the code is actually written, because thinking about testing leads you naturally to create better, more decoupled designs. It helps as you’re writing the code, because it gives you instant feedback on how accurate your code is. And it helps after you’ve written code, both because it gives you the ability to check that the code still works and because it helps others understand how to use your code.

Unit testing is a Good Thing.

But why have a chapter on unit testing in the middle of a book on Ruby? Well, it’s because unit testing and languages such as Ruby seem to go hand in hand. The flexibility of Ruby makes writing tests easy, and the tests make it easier to verify that your code is working. Once you get into the swing of it, you’ll find yourself writing a little code, writing a test or two, verifying that everything is copacetic, and then writing some more code.

Unit testing is also pretty trivial—run a program that calls part of your application’s code, get back some results, and then check the results are what you expected.

Let’s say we’re testing a Roman number class. So far, the code is pretty simple: it just lets us create an object representing a certain number and display that object in Roman numerals:

We could test this code by writing another program, like this:

However, as the number of tests in a project grows, this kind of ad hoc approach can start to get complicated to manage. Over the years, various unit testing frameworks have emerged to help structure the testing process. Ruby comes with Ryan Davis’ MiniTest.[59]

MiniTest is largely compatible with Test::Unit but without a lot of bells and whistles (test-case runners, GUI support, and so on). However, because there are areas where it is different and because there are tens of thousands of tests out there that assume the Test::Unit API, Ryan has also added a compatibility layer to MiniTest. For a little bit more information on the differences between the two, see MiniTest::Unit vs. Test::Unit. In this chapter, we’ll be using the Test::Unit wrapper, because it automatically runs tests for us. But we’ll also be using some of the new assertions available in MiniTest.

MiniTest::Unit vs. Test::Unit

Folks have been using Test::Unit with Ruby for a good number of years now. However, the core team decided to replace the testing framework that comes as standard with Ruby with something a little leaner. Ryan Davis and Eric Hodel wrote MiniTest::Unit as a partial drop-in replacement for Test::Unit.

Most of the assertions in MiniTest mirror those in Test::Unit::TestCase. The major differences are the absence of assert_not_raises and assert_not_throws and the renaming of all the negative assertions. In Test::Unit you’d say assert_not_nil(x) and assert_not(x) ; in MiniTest you’d use refute_nil(x) and refute(x) .

MiniTest also drops most of the little-used features of Test::Unit, including test cases, GUI runners, and some assertions.

And, probably most significantly, MiniTest does not automatically invoke the test cases when you execute a file that contains them.

So, you have three basic options with this style of unit testing:

• require "minitest/unit", and use the MiniTest functionality.
• require "test/unit", and use MiniTest with the Test::Unit compatibility layer. This adds in the assertions inAdditional Test::Unit assertions, and enables the autorun functionality.
• You can install the test-unit gem and get all the original Test::Unit functionality back, along with a bunch of new assertions.

13.1 The Testing Framework

The Ruby testing framework is basically three facilities wrapped into a neat package:

• It gives you a way of expressing individual tests.
• It provides a framework for structuring the tests.
• It gives you flexible ways of invoking the tests.

Assertions == Expected Results

Rather than have you write series of individual if statements in your tests, the testing framework provides a set of assertions that achieve the same thing. Although a number of different styles of assertion exist, they all follow basically the same pattern. Each gives you a way of specifying a desired result and a way of passing in the actual outcome. If the actual doesn’t equal the expected, the assertion outputs a nice message and records the failure.

For example, we could rewrite our previous test of the Roman class using the testing framework. For now, ignore the scaffolding code at the start and end, and just look at the assert_equal methods:

Produces:

The first assertion says that we’re expecting the Roman number string representation of 1 to be “i,” and the second test says we expect 9 to be “ix.” Luckily for us, both expectations are met, and the tracing reports that our tests pass. Let’s add a few more tests:

Produces:

Uh-oh! The second assertion failed. See how the error message uses the fact that the assert knows both the expected and actual values: it expected to get “ii” but instead got “i.” Looking at our code, you can see a clear bug in to_s . If the count after dividing by the factor is greater than zero, then we should output that many Roman digits. The existing code outputs just one. The fix is easy:

Now let’s run our tests again:

Produces:

Looking good. We can now go a step further and remove some of that duplication:

Produces:

What else can we test? Well, the constructor checks that the number we pass in can be represented as a Roman number, throwing an exception if it can’t. Let’s test the exception:

Produces:

We could do a lot more testing on our Roman class, but let’s move on to bigger and better things. Before we go, though, we should say that we’ve only scratched the surface of the set of assertions available inside the testing framework. For example, for every positive assertion, such as assert_equal , there’s a negative refutation (in this case refute_equal ). The additional assertions you get if you load the Test::Unit shim (which we do in this chapter) are listed in Additional Test::Unit assertions, and a full list of the MiniTest assertions is given in Section 13.5, Test::Unit assertions.

The final parameter to every assertion is a message that will be output before any failure message. This normally isn’t needed, because the failure messages are normally pretty reasonable. The one exception is the test refute_nil (or assert_not_nil in Test::Unit), where the message “Expected nil to not be nil” doesn’t help much. In that case, you may want to add some annotation of your own. (This code assumes the existence of some kind of User class.)

Produces:

13.2 Structuring Tests

Earlier we asked you to ignore the scaffolding around our tests. Now it’s time to look at it.

You include the testing framework facilities in your unit test either with this:

or, for raw MiniTest, with this:

Unit tests seem to fall quite naturally into high-level groupings, called test cases , and lower-level groupings, which are the test methods themselves. The test cases generally contain all the tests relating to a particular facility or feature. Our Roman number class is fairly simple, so all the tests for it will probably be in a single test case. Within the test case, you’ll probably want to organize your assertions into a number of test methods, where each method contains the assertions for one type of test; one method could check regular number conversions, another could test error handling, and so on.

The classes that represent test cases must be subclasses of Test::Unit::TestCase. The methods that hold the assertions must have names that start with test. This is important: the testing framework uses reflection to find tests to run, and only methods whose names start with test are eligible.

Quite often you’ll find all the test methods within a test case start by setting up a particular scenario. Each test method then probes some aspect of that scenario. Finally, each method may then tidy up after itself. For example, we could be testing a class that extracts jukebox playlists from a database. (We’re using the low-level DBI library to access the database.)

Produces:

Each test starts by connecting to the database and creating a new playlist builder. Each test ends by disconnecting from the database. (The idea of using a real database in unit tests is questionable, because unit tests are supposed to be fast running, context independent, and easy to set up, but it illustrates a point.)

We can extract all this common code into setup and teardown methods. Within a TestCase class, a method called setup will be run before each and every test method, and a method called teardown will be run after each test method finishes. Let’s emphasize that: the setup and teardown methods bracket each test, rather than being run once per test case. This is shown in the code that follows.

Produces:

Inside the teardown method, you can detect whether the preceding test succeeded with the passed? method.

13.3 Organizing and Running Tests

The test cases we’ve shown so far are all runnable Test::Unit programs. If, for example, the test case for the Roman class was in a file called test_roman.rb, we could run the tests from the command line using this:

Test::Unit is clever enough to run the tests even though there’s no main program. It collects all the test case classes and runs each in turn.

If we want, we can ask it to run just a particular test method:

or tests whose names match a regular expression:

This last capability is a great way of grouping your tests. Use meaningful names, and you’ll be able to run (for example) all the shopping-cart-related tests by simply running tests with names matching /cart/.

Where to Put Tests

Once you get into unit testing, you may well find yourself generating almost as much test code as production code. All of those tests have to live somewhere. The problem is that if you put them alongside your regular production code source files, your directories start to get bloated—effectively you end up with two files for every production source file.

A common solution is to have a test/ directory where you place all your test source files. This directory is then placed parallel to the directory containing the code you’re developing. For example, for our Roman numeral class, we may have this:

This works well as a way of organizing files but leaves you with a small problem: how do you tell Ruby where to find the library files to test? For example, if our TestRoman test code was in a test/ subdirectory, how does Ruby know where to find the roman.rb source file, the thing we’re trying to test?

An option that doesn’t work reliably is to build the path into require statements in the test code and run the tests from the test/ subdirectory:

Why doesn’t it work? It’s because our roman.rb file may itself require other source files in the library we’re writing. It’ll load them using require (without the leading ../lib/), and because they aren’t in Ruby’s $LOAD_PATH, they won’t be found. Our test just won’t run. A second, less immediate problem is that we won’t be able to use these same tests to test our classes once installed on a target system, because then they’ll be referenced simply using require ’roman’.

A better solution is to assume that your Ruby program is packaged according to the conventions we’ll be discussing in Section 16.2, Organizing Your Source. In this arrangement, the top-level lib directory of your application is assumed to be in Ruby’s load path by all other components of the application. Your test code would then be as follows:

And you’d run it using this:

The normal case, where you’re already in the application’s directory, would be as follows:

This would be a good time to investigate using Rake to automate your testing.

Test Suites

After a while, you’ll grow a decent collection of test cases for your application. You may well find that these tend to cluster: one group of cases tests a particular set of functions, and another group tests a different set of functions. If so, you can group those test cases together into test suites , letting you run them all as a group.

This is easy to do—just create a Ruby file that requires test/unit and then requires each of the files holding the test cases you want to group. This way, you build yourself a hierarchy of test material.

• You can run individual tests by name.
• You can run all the tests in a file by running that file.
• You can group a number of files into a test suite and run them as a unit.
• You can group test suites into other test suites.

This gives you the ability to run your unit tests at a level of granularity that you control, testing just one method or testing the entire application.

At this point, it’s worthwhile to think about naming conventions. Nathaniel Talbott, the author of Test::Unit, uses the convention that test cases are in files named tc_xxx and test suites are in files named ts_xxx. Most people seem to use test_ as the test-case filename prefix:

Now, if you run Ruby on the file ts_dbaccess.rb, you execute the test cases in the four files you’ve required.

13.4 RSpec and Shoulda

The built-in testing framework has a lot going for it. It is simple, and it is compatible in style with frameworks from other languages (such as JUnit for Java and NUnit for C#).

However, there’s a growing movement in the Ruby community to use a different style of testing. So-called behavior-driven development encourages people to write tests in terms of your expectations of the program’s behavior in a given set of circumstances. In many ways, this is like testing according to the content of user stories , a common requirements-gathering technique in agile methodologies. With these testing frameworks, the focus is not on assertions. Instead, you write expectations.

Although both RSpec and Shoulda allow this style of testing, they focus on different things. RSpec is very much concerned with driving the design side of things. You can write and execute specs with RSpec well before you’ve written a line of application code. These specs, when run, will output the user stories that describe your application. Then, as you fill in the code, the specs mutate into tests that validate that your code meets your expectations.

Shoulda, on the other hand, is really more focused on the testing side. Whereas RSpec is a complete framework, Shoulda works inside a testing framework, Test::Unit or RSpec. You can even mix Shoulda tests with regular Test::Unit and RSpec test methods.

Let’s start with a simple example of RSpec in action.

Starting to Score Tennis Matches

The scoring system used in lawn tennis originated in the Middle Ages. As players win successive points, their scores are shown as 15, 30, and 40. The next point is a win unless your opponent also has 40. If you’re both tied at 40, then different rules apply—the first player with a clear two-point advantage is the winner.[60]

We have to write a class that handles this scoring system. Let’s use RSpec specifications to drive the process. We install RSpec with gem install rspec. We’ll then create our first specification file:

This file contains nothing more than a description of an aspect of the tennis scoring class (that we haven’t yet written, by the way). It contains a description of the basic scoring system. Inside the description are a set of four expectations (it "should start..." and so on). We can run this specification using the rspec command:[61]

That’s pretty cool. Executing the tests echoes our expectations back at us, telling us that each has yet to be implemented. Coding, like life, is full of these disappointments. However, unlike life, fixing things is just a few keystrokes away. Let’s start by meeting the first expectation—when a game starts, the score should be 0 to 0. We’ll start by fleshing out the test:

Note that we’ve assumed we have a class TennisScorer in a file called tennis_scorer.rb. Our first expectation now has a code block associated with it. Inside that block, we create a TennisScorer and then use a funky RSpec syntax to validate that the score starts out at 0 to 0. This particular aspect of RSpec probably generates the most controversy—some people love it, others find it awkward. Either way, ts.score.should == "0-0" is basically the same as an assertion in Test::Unit.

We’ll beef up our TennisScorer class, but only enough to let it satisfy this assertion:

We’ll run our spec again:

Note that we now have three pending expectations; the first one has been satisfied.

Let’s write the next expectation:

This won’t run, because our TennisScorer class doesn’t implement a give_point_to method. Let’s rectify that. Our code isn’t finished, but it lets the test pass:

Again, we’ll run the specification:

We’re now meeting two of the four initial expectations. But, before we move on, note there’s a bit of duplication in the specification: both our expectations create a new TennisScorer object. We can fix that by using a before stanza in the specification. This works a bit like the setup method in Test::Unit, allowing us to run code before expectations are executed. Let’s use this feature and, at the same time, build out the last two expectations:

Let’s run it:

Finally, RSpec gives us an alternative way of setting up conditions for our tests. The let method creates what looks like a variable (it’s actually a dynamically defined method) whose value is given by evaluating a block. This lets us write the following:

We’re going to stop here, but I suggest that you might want to take this code and continue to develop it. Write expectations such as these:

RSpec has a lot more depth than just the description of expectations. In particular, you can use it with Cucumber, an entire language for describing and running complete user stories. But that’s beyond the scope of this book.

Anyone for Shoulda?

RSpec is testing with attitude. On the other hand, Shoulda takes many of the ideas from RSpec and humbly offers them to you for integration into your regular unit tests. For many developers, particularly those with existing Test::Unit tests, this is a good compromise. You get much of the descriptive power of RSpec-style expectations without having to commit to the full framework.

Install Shoulda using gem install shoulda. Then, unlike RSpec, write a regular Test::Unit test case. Inside it, though, you can use the Shoulda mini-language to describe your tests.

Let’s recast our final RSpec tennis scoring tests using Shoulda:

Behind the scenes, Shoulda is creating Test::Unit test methods for each should block in your tests. This is why we can use regular Test::Unit assertions in Shoulda code. But Shoulda also works hard to maintain the right context for our tests. For example, we can nest contexts and their setup blocks, allowing us to have some initialization that’s common to all tests and some that’s common to just a subset. We can apply this to our tennis example. We’ll write nested contexts and put setup blocks at each level. When Shoulda executes our tests, it runs all the appropriate setup blocks for the should blocks.

Let’s run it:

Would we use these nested contexts for this tennis scoring example? We probably wouldn’t as it stands, because the linear form is easier to read. But we use them all the time when we have tests where we want to run through a complex scenario that builds from test to test. This nesting lets us set up an environment, run some tests, then change the environment, run more tests, change it again, run even more tests, and so on. It ends up making tests far more compact and removes a lot of duplication.

13.5 Test::Unit assertions

assert | refute(boolean, <message> )

Fails if boolean is (is not) false or nil.

assert_block { block }

Expects the block to return true.

assert_ | refute_ empty(collection, <message> )

Expects empty? on collection to return true (false).

assert_ | refute_ equal(expected, actual, <message> )

Expects actual to equal/not equal expected, using ==.

assert_ | refute_in_delta(expected_float, actual_float, delta, <message> )

Expects that the actual floating-point value is (is not) within delta of the expected value.

assert_ | refute_ in_epsilon(expected_float, actual_float, epsilon=0.001, <message> )

Calculates a delta value as epsilon * min(expected, actual) and then calls the _in_delta test.

assert_ | refute_ includes(collection, obj, <message> )

Expects include?(obj) on collection to return true (false).

assert_ | refute_ instance_of(klass, obj, message )

Expects obj to be (not to be) a instance of klass.

assert_ | refute_ kind_of(klass, obj, <message> )

Expects obj to be (not to be) a kind of klass.

assert_ | refute_ match(regexp, string, <message> )

Expects string to (not) match regexp.

assert_ | refute_ nil(obj, <message> )

Expects obj to be (not) nil.

assert_ | refute_ operator(obj1, operator, obj2, <message> )

Expects the result of sending the message operator to obj1 with parameter obj2 to be (not to be) true.

assert_raises(Exception, ...) { block }

Expects the block to raise one of the listed exceptions.

assert_ | refute_ respond_to(obj, message, <message> )

Expects obj to respond to (not respond to) message (a symbol).

assert_ | refute_ same(expected, actual, <message> )

Expects expected.equal?(actual).

assert_send(send_array, <message> )

Sends the message in send_array[1] to the receiver in send_array[0], passing the rest of send_array as arguments. Expects the return value to be true.

assert_throws(expected_symbol, <message> ) { block }

Expects the block to throw the given symbol.

flunk(message="Epic Fail!")

Always fails.

skip(message)

Indicates that a test is deliberately not run.

pass

Always passes.

Additional Test::Unit assertions

assert_not_equal(expected, actual, <message> )

Expects actual not to equal expected, using ==. Like refute_equal .

assert_not_match(regexp, string, <message> )

Expects string not to match regexp. Like refute_match .

assert_not_nil(obj, <message> )

Expects obj not to be nil. Like refute_nil .

assert_not_same(expected, actual, <message> )

Expects !expected.equal?(actual). Like refute_same .

assert_nothing_raised(Exception, ...) { block }

Expects the block not to raise one of the listed exceptions.

assert_nothing_thrown(expected_symbol, <message> ) { block }

Expects the block not to throw the given symbol.

assert_raise(Exception, ...) { block }

Synonym for assert_raises .

Footnotes