Selenium Design Patterns and Best Practices (2014)
Chapter 3. Refactoring Tests
"A bear, however hard he tries,
Grows tubby without exercise...."
--A. A. Milne, "Teddy Bear" from The Complete Poems of Winnie-the-Pooh
Exercise is an important part of keeping your body fit; however, it can be easily despised. Exercising takes a lot of work, causes a lot of physical pain, and gives very few instant results to keep you motivated. Our test suite needs some upkeep and refactoring to remain in pristine condition.
In the previous chapter, we wrote two tests but we did it in a quick and sloppy manner. A lot of code was duplicated, and simply copied and pasted. If we keep growing our suite in a similar manner, it will become unmanageable in no time! In this chapter, we will put our test suite on a treadmill to get it into a better overall shape. We will cover the following topics:
· Refactoring tests
· The DRY principle
· The DRY testing pattern
· Setup and teardown methods
· The Hermetic test pattern
· Test independence
· Using timestamps to create unique test data
· Sharing common functionalities between tests
· Random run order practice
Since this chapter will be focused on refactoring tests, let's first define this term. Refactoring is the act of restructuring your code to improve the internal efficiency, stability, and long-term maintainability without adding or modifying any of the underlying functionalities. At the end of the refactoring session, we should not have any new tests; the only goal is to improve the existing tests.
Since there are no obvious instant results such as having 10 more new tests, refactoring may seem like a waste of time. However, having two tests that do not randomly fail is a lot more productive in the long run than having 12 tests that cannot be relied on. Refactoring your tests is similar to calisthenics; if you don't exercise, you will probably die of a heart attack 20 years before your time. That being said, we will not add any new tests in this chapter. Instead, we will improve the product review tests in Chapter 2, The Spaghetti Pattern.
The DRY testing pattern
Treating automated tests with the same care and respect as the application that we are trying to test is the key to long-term success. Adopting common software development principles and design patterns will prevent some costly maintenance in the future. One of these principles is the Don't Repeat Yourself (DRY) principle; the most basic idea behind the DRY principle is to reduce long-term maintenance costs by removing all unnecessary duplication.
There are a few times when it is okay to have a duplicate code, at least temporarily. As Donald Knuth so eloquently stated, "Premature optimization is the root of all evil (or at least most of it) in programming."
The DRY testing pattern embraces the DRY principle and expands on it. Not only do we remove the duplicate code and duplicate test implementations, but we also remove duplicate test goals.
A test goal or test target is the main idea behind any given test. The rule of thumb is if you cannot describe what the test is supposed to accomplish in a sentence or two, then the test is too complicated or it does not understand what it is testing. A good example of a test goal would be "an anonymous user should be able to purchase product X on the website."
We are trying to avoid accidentally testing any functionality not related to the current test. For example, if the target of the current test is the registration flow, this test should not fail if a social media icon fails to load. Social media icons should have a test of their own that is not related to registration tests.
David Thomas and Andrew Hunt formulated the DRY principle in their book, The Pragmatic Programmer, by Andrew Hunt and David Thomas, published by Addison-Wesley Professional. The DRY principle is sometimes referred to as Single Source Of Truth (SSOT) or Single Point Of Truth (SPOT) because it attempts to store every single piece of unique information in one place only.
Advantages of the DRY testing pattern
Writing tests using the DRY testing pattern has many advantages. Here are four advantages:
· Modular tests: Tests and test implementations are self-sufficient. Any test can run in any order. Also, test actions such as clicking or registering a new user are shared during the tests.
· Reduced duplication: All actions such as filling out a form are neatly kept in a single place instead of having multiple copies peppered all over the suite.
· Fast updates: Having unique actions in a single place makes it easy to update the tests to mimic new growth of the application.
· No junk code: Constant upkeep of the test suite, with deletion of duplicates, prevents the test suite from having code that is no longer used.
Disadvantages of the DRY testing pattern
There are some disadvantages of setting your tests according to the DRY testing pattern; it is a lot of work and requires a lot of buy in from the whole team. Here are some of the most common issues:
· Complicated project structure: Some test actions will be logically grouped with other similar actions. Filling out a login form and clicking the login button will probably happen in the same implementation file. However, some actions will inevitably end up in a different file, making it hard to find them.
· Lack of a good IDE: There aren't many good IDEs that will notify the test developer if a test action has already been implemented. Most test developers will reimplement the action they need instead of looking for it.
The best way around this problem is to have a lot of in-team communication. Asking whether anyone has already implemented an action in code will save you time and prevent duplication.
· Constant upkeep: Keeping the test suite clean and applying the DRY test pattern will need dedication from the team. Duplicate code needs to be pruned and deleted instead of being ignored.
In statically typed languages such as Java, we can use static analysis tools that can be used to monitor code duplication.
· Programming skills: This needs to be improved by the whole team. One test developer who keeps duplicating logic and cargo-culting can spoil the elegant test suite in a matter of weeks.
Cargo cult is a phrase commonly used to describe a programming style where a programmer uses a piece of code without understanding what the original intention of that code was. It can be described as "we do this because we always did this; I don't know why."
Let's start DRY-ing out our tests. The first and obvious choice is the setup and teardown methods.
Moving code into a setup and teardown
Most modern testing frameworks include the concept of a setup and teardown. Each framework can call them by a different name. For example, in Cucumber, the setup is called Background, while in Rspec, it is called Before. No matter what name the framework chooses, the idea behind these two methods remains the same. This setup is run before tests and is used to get the environment in a test-ready state. The teardown is used to clean up after the tests to put the environment back into a pristine state. Some frameworks allow the setup and teardown to be run before and after each individual test, while others only allow them to be executed before and after a group of tests; some even allow both.
Let's start cleaning up product_review_test.rb from Chapter 2, The Spaghetti Pattern, by adding the setup and teardown methods:
You can find the complete product_review_test.rb file at http://awful-valentine.com/code/chapter-2/.
1. The first thing we will do is add the setup and teardown methods at the top of our test; our code will look like this:
2. Let's move the creation of the Firefox instance into setup and the quitting of the Firefox instance into teardown, as seen in the following screenshot:
We changed all instances of the selenium variable to @selenium; this makes our variable an instance variable for the current test class. Individual tests are now able to reference the @selenium variable instead of having to create their own.
Our tests are starting to look better right away; the setup method is helping us to create a new instance of Firefox before each test is started. The biggest advantage is that the teardown method will execute every single time the test finishes. This means that Firefox will be closed every time, even if the test fails before completion.
Removing duplication with methods
Let's keep refactoring our test's logic; the next item to refactor is the click on the home page for the product we desire to comment on. Let's create a new method called select_desired_product_on_homepage, and move the click code inside it, as seen here:
After we move the click action to the new method, we need to invoke this method from our test, like this:
We need to perform the same refactoring in the test_adding_a_duplicate_review test. This way, both tests use the same method call to select a product on the home page.
Removing external test goals
In the previous chapter, we added two assertions; they were there to verify that clicking on the MORE INFO link on the home page takes us to the correct product. The assertions are shown here:
These do not adhere to the DRY testing pattern, because the tests are no longer testing the product review functionalities; now they also test the product retrieval and display functionalities.
Besides performing duplicate assertions, they make the test suite unstable. Test that check product review functionality broke because of a minor editorial change to the page content. The most logical thing to do is delete this instability causing code and write a set of tests specifically to test product descriptions. We will delete the assertions now and add the new tests in Chapter 4, Data-driven Testing.
Let's take a look at the test after we delete the unnecessary assertions:
Using a method to fill out the review form
Finally, the biggest chunk of duplication comes from filling out the review form. Since the values that we insert in the form fields are identical between the two tests, we should be able to easily pull out this duplication to fill_out_comment_form. Our new method will look like this:
Reviewing the refactored code
Our tests are starting to look completely different from when we first began refactoring. In the book Refactoring to Patterns, the author Joshua Kerievsky, shows readers how to refactor code into a series of small, easy-to-understand actions. The idea is to avoid refactoring the whole class or file in one go. We might have an idea of how the code is supposed to look when we are finished with it, but rewriting everything into its final form right away tends to prove difficult. So, it is better to take many tiny steps than one giant step that will often leave us confused and frustrated.
Refactoring to Patterns by Joshua Kerievsky, published by Addison-Wesley Professional.
Staying with the principle of small, easy-to-understand changes, let's review our test code so far before we make any more modifications to it. First, let's take a look at the four new methods we added.
Everything should look familiar so far. The setup and teardown methods run before and after each test and manage the instances of Selenium WebDriver and Firefox. The select_desired_product_on_homepage method can be invoked on the home page to click on the MORE INFObutton for the product we want to review. Finally, the fill_out_comment_form method fills out the review form for the product we've selected.
Next, let's take a look at test_add_new_review, shown here:
So far, we have dramatically improved the code quality of our tests. By moving out some of the duplication into individual methods, we made our test suite a lot easier to maintain in the long run. The select_desired_product_on_homepage and fill_out_comment_form methods can now be reused by any test in our test class. This means that if we ever need to update our test to adhere to the new functionalities, we only need to do it once in the appropriate method; all of the tests will automatically work.
Since we are extremely dedicated to having a good test suite, we will not stop refactoring just yet. Our next goal is to fix test instability caused by the Spaghetti pattern; we will break the test-on-test dependency in the next section.
Make sure you fully understand all of the actions performed so far before moving on to the next section.
The Hermetic test pattern
The Hermetic test pattern is the polar opposite of the Spaghetti pattern; it states that each test should be completely independent and self-sufficient. Any dependency on other tests or third-party services that cannot be controlled should be avoided at all costs. It is impossible to get a perfectly hermetically sealed test; however, anytime a dependency on anything outside the test is detected, it should be removed as soon as possible.
The Hermetic test pattern can also be referred to as Test is an Island Pattern, which is a play on the word from an old saying no man is an island.
Advantages of the Hermetic test pattern
The Hermetic pattern is especially appealing when trying to flush out test instability. Here are some advantages to hermetically seal your tests:
· Clean start: Each test has a cleaned up environment to work in. This prevents accidental test pollution from previous tests, such as a created user that should not be present.
· Resilience: Each test is responsible for its own environment, so it has no need for everything to go perfectly right somewhere else in the suite.
We will talk about data management for individual tests in Chapter 4, Data-driven Testing.
· Modular: Each test is standalone and can be rearranged into smaller test suites such as a smoke suite, or can run as a standalone test.
Smoke suite refers to a set of smoke tests, which is run on any environment, usually immediately post deploy of a new version. The idea is to quickly smoke out any issues in the new build. Other types of test suites are discussed in Chapter 8, Growing the Test Suite.
· Random run order practice: Since tests do not depend on successful completion of any other test in the suite, they can be executed in any order. We'll cover more on random run order practice later in this chapter.
· Parallel testing: If our test suite can run in any random execution order, it can also be executed in parallel. Executing multiple tests simultaneously can significantly reduce the runtime of the test suite.
Disadvantages of the Hermetic test pattern
Hermetically sealing each test comes with an increase in individual test stability but it does have some disadvantages:
· Upfront design: Each test needs to be designed to be self-sufficient. Each test can reuse methods used by other tests, but cannot reuse data and test results generated by other tests.
· Runtime increase: Since each test has to set up the environment before it starts, the runtime of each individual test is increased.
This effect is easily negated when the test suite is executed in parallel.
· Resource usage increase: Increased runtime of individual tests means that the test suite will need more resources, such as RAM, to run tests in parallel.
Removing test-on-test dependence
In the previous chapter, we created two tests; one of which depended on the success of the first test for the environment to be in a testable state. So far, we removed some of the duplication from our tests but we did not solve the data interdependency. You may have probably noticed that running our tests multiple times in a row will fail tests in unpredictable ways unless you constantly update the test data manually in between the test runs. In this section, we will make our tests independent of hardcoded test data and each other.
Our website only checks that the content of the comment is unique; it does not check whether any of the other fields such as name are duplicates. So, we can modify our test to provide a new comment every time the fill_out_comment_form method is called by passing in the comment we want to add:
Our fill_out_comment_form method now accepts comment text from the test, so we can make our test generate a unique comment and pass it to the fill_out_comment_form method.
Using timestamps as test data
Using timestamps to guarantee unique data is a great shortcut when writing tests, since it is very unlikely that two tests will be executed at the exact same time. Let's create a method that will generate a unique comment by adding a timestamp to it, as follows:
In Ruby, the last statement in a method is automatically returned; so, having return is redundant in this case.
What makes our generate_unique_comment method work is the Time.now.to_i call, which returns a timestamp in seconds since the epoch.
Epoch is a UNIX timestamp of seconds elapsed since January 1, 1970.
If we were to print out our unique comment with the timestamp, it would look something like this:
Let's add a new variable to our test, which will generate and store the unique comments, so that we can use it later in the test when we perform an assertion. Our code will look like this with the new variable pointed out with arrows:
Extracting the remaining common actions to methods
Now that we got the pesky data uniqueness out of the way, we can refactor our tests even further! Currently, our test_add_new_review test fills out a product review every single time it runs. At the same time, our test_adding_a_duplicate_review test relies on that review to exist before the actual assertion can take place. Since both tests are using the same functionalities to create a product review, we can extract and reuse duplicate code between the tests by extracting it to a method and having both tests call the said method.
Creating a new review with a single method call
Here are the four actions that both of our tests share:
· Navigating to the home page: The starting point of both of the tests is to get to the home page. The rest of the execution starts from this point.
· Select product: A given product needs to be selected and the MORE INFO button needs to be clicked, so that we can review the product.
· New review form completion: Both tests fill out the product review form with identical data.
· Retrieve new review ID: After the review is created, the review ID is retrieved for double assertion.
This step is optional for the duplicate review test, since it does not actually do anything with the review ID.
We need to create a new method that can easily be called by both the tests. To make the generate_new_product_review method portable and reusable, we will not implement the previously described tests in it. Instead, we will create several helper methods that thegenerate_new_product_review method will call. First, let's create a method to navigate to the home page, as shown here:
We already have the methods to select a product on the home page and fill out the review form; so we will skip those. Let's make a method to retrieve the newly generated review ID now; get_newly_created_review_id is shown as follows:
Let's pull together all of these methods into generate_new_product_review like this:
As mentioned earlier, Ruby implicitly adds the return statement to the very last action in any method. This means that the generate_new_product_review method will automatically give the returned value of the get_newly_created_review_id method, which happens to be the new review comment.
Let's take a look at test_add_new_review after we finish this round of refactoring. It will look like this:
Our test now generates a unique comment, passes that comment to review a generation method, and uses the returned review_id to collect the needed information for assertions. Let's take a look at test_adding_a_duplicate_review now:
The duplicate review test benefits the most from this refactoring. At this point, it has to call the generate_new_product_review method two times in a row to be ready for assertions. This test is now completely independent. Even if test_add_new_review never runs, it will still be able to test duplicate review functionalities.
Note that we added sleep 10 (denoted with an arrow) in our duplicate review test. The product review form has another fraud detection mechanism, which prevents users from posting product reviews too rapidly. The sleep 10 call will allow 10 seconds to elapse between review creations to work around this limitation.
Reviewing the test-on-test dependency refactoring
As we did before, let's review our refactoring session progress. The setup and teardown methods have not changed at all.
The order of method declarations has changed to make the test class more aesthetically pleasing to read.
The two tests for the product review functionalities look like this:
Finally, all of the refactored out helper methods are moved into the private section of the test class, as follows:
Creating generic DRY methods
At this point, our test is no longer recognizable compared to what it was at the beginning of the chapter. Before we wrap it up, let's talk about the generic action used all through our tests. Throughout the test code, we use some common methods to perform actions such as clicking or typing text into a text field. These chained methods look something like this:
What if we refactor these methods a little further and create some generic private methods that can be used in a much simpler way? Let's start with the most common method used, @selenium.find_element, and create a generic find_element method:
Our find_element method now accepts an element identifier and an optional strategy parameter. If the strategy is not specified, it will default to the CSS selector.
More information about element locator strategies can be found in Chapter 2, The Spaghetti Pattern.
Now let's add two new methods that use our local find_element method to click and type text into text fields. These methods look like this.
Refactoring with generic methods
Now that we have three very generic methods that allow tests to interact with the web page, you have a small homework assignment. Finish refactoring the rest of the test, and see how many more generic methods you can create. I'll give you a hint; the fill_out_comment_form method from earlier now looks something like this:
We will take greater advantage of generic action methods when we get to the The Action Wrapper pattern section in Chapter 5, Stabilizing the Tests.
Before we finish this chapter, let's take a look at the random run order principle, which takes advantage of the refactoring we just completed. Without all of this work, we would never be able to put this principle into practice.
The random run order principle
Random run order is more of a principle than a pattern. It applies to the test execution. This execution is usually performed on a Continuous Integration (CI) environment. The random run order principle states that the order of the test suite execution should be randomized every time the suite is executed. The idea is to flush out instabilities in the test suite by introducing an element of chaos. Any test that has a hidden dependency on another test will eventually fail when the test run order is random.
CI tools are simple applications that execute a given build when certain conditions such as code change are met. There are several commercial tools available, such as TeamCity, Bamboo, and Travis-CI. One of the most popular and free open source CI tools is Jenkins, which can be found at http://jenkins-ci.org/.
Advantages of the random run order principle
Let's talk about the advantages of running our tests in a random order:
· Prevents test interdependence: Any test that depends on another test to set up the environment will be exposed rather quickly. This forces us to maintain good Hermetic integrity within each test.
The type of test-on-test dependency described here is the Chain Linked pattern, discussed in Chapter 2, The Spaghetti Pattern.
· Flushes out data pollution: Sometimes test flakiness does not come from the setup stage of the test. Often we will find a test that intentionally puts the test environment into a bad state on purpose, to test the application resilience. If the said test leaves the environment in a bad state after completion, it can break the test that follows.
Ideally, each test will set up the test environment in the setup stage of the execution, and return the environment to the original state every single time in the teardown stage.
· Built in: Some test frameworks not only support test randomization out of the box, but also have the random run order as a default setting.
Disadvantages of the random run order principle
As always, there are some negatives when making your test suite compatible with the random run order; here is a short list:
· A lot of refactoring: If we have a large and mature test suite with copious usage of the Spaghetti pattern and the Big Ball of Mud pattern, making tests compatible with a random run order is tremendous amount of work.
· Random run audits: If the test run is completely random for every single build, it can be difficult to know the sequence of tests that causes the instability. One way to solve this difficulty is to have an audit trail for each build that will allow us to know the exact sequence of tests that led us to failure.
· Team frustration: Running tests in a random order can create a lot of frustration and resentment for the whole team. When working on a deadline, having your build fail due to unrelated data pollution is annoying.
Remember that to go fast, one needs to start slow. Issues like these will always slow down the whole process initially. Once they are exposed and fixed, the overall velocity of the test development actually goes up.
· No built-in support: A lot of test frameworks do not support test randomization. Implementing this functionality might be very difficult.
In this chapter, we discussed the harmful effects of code duplication on the test suite. As the application keeps evolving, the time required to keep the tests up to date grows exponentially. The solution for this is to avoid code and test duplication by using the DRY testing pattern. We applied the DRY principle by refactoring duplicate code into the setup and teardown methods and other methods that our tests can share.
We also removed the interdependency between our two tests by using the Hermetic test pattern. Removing the Spaghetti pattern from our suite has dramatically increased the test stability. Random order ability was achieved by hermetically sealing our tests and having them use unique data.
In the next chapter, we will be concentrating on test data and how to manage it in different environments.