Testing with F# (2015)

Chapter 6. Functional Testing

Functional testing involves testing the function of a program. It's a higher form of abstraction than the testing we have done so far, as it is a black box test and we don't care about the internal workings of our program, only that it delivers the expected results.

In this chapter, we will learn about the following topics:

· What functional testing is and how to apply it

· Writing executable specifications with TickSpec

· Wielding a web browser as a testing tool

· Which tests provide the best regression for our application

Functional testing is a practice with many names and is often confused with manual testing. This is because what testers do is functional testing, even though they are doing it manually. In order to ensure the quality of the system, testers will verify the functionality with the specification. I've heard of testers that use automated tests to ensure they have regression in their testing, without having to repeat the same testing task over and over again. This makes functional testing not only a development task, but also a common task for testers and developers alike.

Specifications

I hear very often when coaching teams that, we are doing agile so we don't write specifications.

One of the most common misconceptions around agile testing is that you do not create any documentation. This is because the main principal working software for comprehensive documentation is often misinterpreted as "do not write documentation." They're missing the whole point of the agile manifesto, where there is value in the items on the right; however, we value the items on the left more.

Agile testing is striving for a learning process in which you can adapt to it through a software development process. The error in the waterfall procedure is in trying to specify everything from the beginning because we can't know everything from the start. The domain is too complex, and we need to break it down into chunks and start with the most important ones. Once we've done this, we will also have learned new things that will let us re-evaluate how to tackle the rest of the problem, and this is true until the project ends.

Specifications are crucial for development in order to know when you're done, but they are also crucial for functional testing because the specification will be your answer if the feature is correctly implemented. You do not specify the whole system at the start of the project. Instead, you should have a dedicated tester on your team that will specify the most prioritized product backlog items so they're ready when it's time for sprint planning. This will also make committing user stories to the sprint much easier.

Before the work is begun, the functionality is specified. After the work has been completed, the functionality is verified according to its specification. And after the functional tests have been written, the specification can be thrown away.

Setting up TickSpec

TickSpec is a lightweight Behavior-driven Design (BDD) framework for F#, influenced by SpecFlow from C#, which has its root in the Ruby testing framework, Cucumber. What is common between all these frameworks is that they implement a domain specific language called Gherkin, which is used to express the tests.

Let's start by adding TickSpec from the NuGet package manager to our application, as shown in the following screenshot:

Next up, we will write our first feature file. This file will specify what we expect of the feature and provide scenarios for us to verify. This file is written in Gherkin DSL to mimic the natural language.

Here is a feature file for Conway's Game of Life:

Feature: Conway's Game of Life

Scenario 1: Any live cell with fewer than two live neighbors dies, as if caused by under-population.

Given a live cell

And has 1 live neighbor

When turn turns

Then the cell dies

Scenario 2: Any live cell with two or three live neighbors lives on to the next generation.

Given a live cell

And has 2 live neighbors

When turn turns

Then the cell lives

Scenario 3: Any live cell with more than three live neighbors dies, as if by overcrowding.

Given a live cell

And has 4 live neighbors

When turn turns

Then the cell dies

Scenario 4: Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.

Given a dead cell

And has 3 live neighbors

When turn turns

Then the cell lives

The magic words in each scenario are Given, When, and Then. This structure set up the user case we wanted to test. They were also mapped to definitions written in F# that specify how each of the Given, When, and Then words should be interpreted.

Even though the code definition looks like plain English, it is written in a format we have decided will be easy for F# definitions to interpret and lend itself well for reuse.

Let's look at the code definitions:

module GameOfLifeDefinitions

open TickSpec

open NUnit.Framework

open Microsoft.FSharp.Reflection

open GameOfLife

let mutable cell = Dead(0, 0)

let mutable cells = []

let mutable result = []

let [<Given>] ``a (live|dead) cell`` = function

| "live" -> cell <- Live(0, 0)

| "dead" -> cell <- Dead(0, 0)

| _ -> failwith "expected: dead or live"

let [<Given>] ``has (\d) live neighbors?`` (x) =

let rec _internal x =

match x with

| 0 -> [cell]

| 1 -> Live(-1, 0) :: _internal (x - 1)

| 2 -> Live(1, 0) :: _internal (x - 1)

| 3 -> Live(0, -1) :: _internal (x - 1)

| 4 -> Live(0, 1) :: _internal (x - 1)

| _ -> failwith "expected: 4 >= neighbors >= 0"

cells <- _internal x

let [<When>] ``turn turns`` () =

result <- GameOfLife.next cells

let [<Then>] ``the cell (dies|lives)`` = function

| "dies" -> Assert.True(GameOfLife.isDead (0, 0) result, "Expected cell to die")

| "lives" -> Assert.True(GameOfLife.isLive (0, 0) result, "Expected cell to live")

| _ -> failwith "expected: dies or lives"

Here, we find the words Given, When, and Then again. Instead, it specifies how the keyword should be interpreted. In the first example, we mapped the following:

· Given a live cell

· Given a dead cell

In the function names, we allow regular expressions in order to match more than one exact statement from the specification. So, when we create a match such as the [<Given>] ``a (live|dead) cell`` function, it will match both the Given statements from the specification.

Using regular expressions, we can use the same implementation for both live and dead values, and this makes the definition reusable.

In the next example, we will match the number of live numbers that are given. This will quickly reveal why I chose to write a number instead of the words one, two, or three, which would sound more like natural English, but be harder to interpret.

The function recursively builds the test domain by calling itself with a decremented number of neighbors every time.

The When clause is meant to execute the test the same way as in our previous test pattern. Then asserts that the result was expected.

Next, we need to tie it together with NUnit so it becomes a proper test. This comes directly from the example code by Phil Trelford that is installed together with the TickSpec package:

module NUnit.TickSpec

open NUnit.Framework

open System.IO

open System.Reflection

open TickSpec

let assembly = Assembly.GetExecutingAssembly()

let definitions = new StepDefinitions(assembly)

/// Inherit from FeatureFixture to define a feature fixture

[<AbstractClass>]

[<TestFixture>]

type FeatureFixture (source:string) =

[<Test>]

[<TestCaseSource("Scenarios")>]

member this.TestScenario (scenario:Scenario) =

if scenario.Tags |> Seq.exists ((=) "ignore") then

raise (new IgnoreException("Ignored: " + scenario.Name))

scenario.Action.Invoke()

member this.Scenarios =

let s = File.OpenText(Path.Combine(@"..\..\",source))

definitions.GenerateScenarios(source,s)

type Feature () = inherit FeatureFixture("GameOfLifeFeature.txt")

This will generate NUnit tests for the scenarios. After compiling the tests, you will see the following in your Test Explorer window (or any other chosen test runner):

These tests will now fail until we have implemented the actual Game of Life. This implementation could look something like this:

module GameOfLife

type Cell = | Dead of x : int * y : int | Live of x : int * y : int

// get coordinate from a cell

let coord = function | Live(x, y) -> x, y | Dead(x, y) -> x, y

// get if the cell on the coordinate is live

let isLive coordinate cells =

cells |> List.exists (fun cell -> cell |> coord = coordinate)

// get if the cell on the coordinate is dead

let isDead coordinate cells =

not (isLive coordinate cells)

let live = function Live(_,_) -> true | _ -> false

let dead = function Dead(_,_) -> true | _ -> false

// get all surrounding cells

let neighbors (cell : Cell) cells =

let x, y = cell |> coord

let neighbors = [(x, y - 1); (x, y + 1); (x - 1, y); (x + 1, y)]

neighbors

|> List.map(fun coord ->

match isLive coord cells with

| true -> Live(coord)

| false -> Dead(coord))

// is list length less than n

let moreThanOneLessThanThree (list : 'a list) =

list.Length > 1 && list.Length < 4

// get the next play board

let next (cells : Cell list) =

printfn "%A" cells

// Scenario 1: Any live cell with fewer than two live neighbors dies, as if caused by under-population.

// Scenario 3: Any live cell with more than three live neighbors dies, as if by overcrowding.

let oldCells =

cells

|> List.filter (fun cell -> (neighbors cell cells) |> List.filter live |> moreThanOneLessThanThree)

// Scenario 4: Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.

let newCells =

cells

|> List.collect(fun cell -> neighbors cell cells)

|> List.filter dead

|> List.filter (fun cell -> neighbors cell cells |> List.filter live |> List.length |> ((=) 3))

|> List.map (fun cell -> Live(cell |> coord))

oldCells @ newCells

This is one way to implement Conway's Game of Life, but there are many implementations out there that are much more efficient than this one.

Executable specifications

The executable specification is somewhat a mystery, as it often falls between the chairs. It's not picked up by developers because it operates on an abstraction level that is not attractive to them. It also doesn't work as a design tool as unit testing does.

It may, however, be a very useful tool for any tester that is ready to write some code. Sure, the executable specification reads well in plain English, but you will never have product owners writing executable specifications that will just work or even be the base for the implementation.

Instead, the most useful task for executable specifications is for the product owner and team to find a common ground where they can talk about such specifications. The product owner will drive the requirements for the specifications of the functionality, and the developers will put these in a format that can be made into executable specifications. This is where you will find that executable specifications become a very powerful collaboration tool.

In order to get a good test suite, you will have to write high-quality feature files. There are some criteria you should consider when working on feature files:

· Let it be about the feature and not about the syste.

· It's a specification, and not a test script

· A scenario should test only one thing

· It should be short and concise

· Scenarios should be easy to access

You might want to go to the feature files in order to find out what is expected of a feature. This puts very high requirements on the quality of those feature files. If they are full of repetition or are long and tedious, they will not be used as the main source for the specification documentation.

The typical anti-patterns of feature files are as follows:

· Long sentences

· Repetitive sentences or scenarios

· Repetitive use of "and" to check just one more thing

Try to write the feature files as if they were written in plain English, even if there is a format enforced. With this, we should be able to write feature files that are easy and fun to read, and we could begin by reading them to ourselves in order to achieve this.

Scenarios that are the same from one another with just a few variables swapped out are not easy to read and will not be used as active documentation. If there is repetition, one can use a table in order to bake those scenarios into one:

Feature: Contact form

Scenario: Both e-mail fields must add up

Given I have entered the following into the contact form

| Name | Email1 | Email2 | Phone | Message |

| Mikael | hello@test.com | hey@test.com | 5551234 | Hello |

When submitting the form

Then I should be informed of mismatching e-mail addresses

Even if very few definitions in this feature will be reusable, it reads better to put data like this in a form.

When you have a lot of setup to do for these scenarios, they can be expressed as the background of your feature:

Background:

Given a global administrator named "Greg"

And a blog named "Greg's anti-tax rants"

And a customer named "Dr. Bill"

And a blog named "Expensive Therapy" owned by "Dr. Bill"

This example, taken from the SpecFlow documentation, shows how to properly set up the prerequisites for the scenarios, removing this repetition from each scenario in the feature file.

Of course, the great thing about executable specifications that we never must forget is that its documentation will never become obsolete. Once you change the feature without updating the specification, the tests will break.

Combinatorial examples

One of the more powerful features of executable specifications is the combinatorial example, which lets you use examples as input for your test. In the same manner as NUnit test cases, this allows you to write one test case and verify it with a good amount of different data:

Feature: Sign up form

Scenario: E-mail registration should only accept valid addresses

Given I've entered my name and agreed to terms and conditions

When I enter <email> into the e-mail field

Then the page should let me know the registration was <success>

Examples:

| email | success |

| hello@mikaellundin.name | true |

| hello.you@mikaellundin.name | true |

| hey@litemedia.se | true |

| invalid | false |

| @mikaellundin.name | false |

| hello.you.mikaellundin.name | false |

| hey@litemedia | false |

Here, we wanted to verify a registration form on a web page by sending in a couple e-mail addresses. We did not specify what the system would do depending on the result, only that the system would give some kind of feedback through which we would determine whether the e-mail address was accepted or not. Our test isn't really affected if we get an error page or error message by AJAX.

Here, I have implemented the definitions file to match the examples and pick them up in our tests:

module SignUpDefinitions

open TickSpec

open NUnit.Framework

let [<Given>] ``I've entered my name and agreed to terms and conditions`` () =

// shortened for brewity: setup

()

[<When("I enter {0} into the e-mail field", "(.*?)")>]

let [<When>] ``I enter (.*?) into the e-mail field`` (email : string) =

// shortened for brewity: write the e-mail into the field on the page

printf "%A" email

[<Then("the page should let me know the registration was {0}", "(true|false)")>]

let [<Then>] ``the page should let me know the registration was (true|false)`` (success : bool) =

// shortened for brewity: verify what the system did after entering e-mail

if success then

Assert.True(success);

else

Assert.False(success);

The first thing you will notice as strange is the double attributes on When/Then. The first attribute will match the data from the table with the test, and in that sense, generate the tests that will run.

The second attribute is the same as our previous example: mapping the input data to the arguments of the function.

When we run this, it almost looks like magic:

This is the essence of data-driven testing, where we can have the data expressed in a readable format and pushed through our tests without much setup.

Web-browser-based testing

When working with web-based interfaces, the best way to execute functional testing is to query the website and assert on the result HTML. There are some tools and methods to go about doing this, and we will now take a look at a few of them.

Before starting, it is worth discussing where this testing will take place. Before you can use a web client to functionally test a website, you need to deploy this website somewhere. You also need to set up the website with a predictable state so the test doesn't fail because an editor has moved around some content.

How to set up a web-based application is very targeted to the application. Often, you will base a new website on Content Management System (CMS), and this system has its own methods to set up a particular state. Sometimes, however, it is enough to roll back a database backup before the test suite runs, and at other times, you need to create a remote procedure that will set up the necessary state for you. At any rate, predictability is key when it comes to automating functional testing.

When working with long-running flows, such as a shopping cart checkout, it will become necessary for you to create an API that will allow your tests to set the current session. If you're also doing manual testing, this could be a web page where one could add any product and promotion with a click of a button instead of the normal flow. If you don't have a manual tester, a remote procedure call might also be suitable to set the state. This will allow your test to start in the middle of the order checkout without the long setup. Experience tells us that long setups in a web client-base testing environment have a tendency to fail and bring a huge part of your test suite with it.

Just make sure you don't ship your application together with the test harness that allows you to directly manipulate the session, as this would be a huge security concern.

Selenium

Selenium has been the most obvious tool to run web browser-based testing for years. It works by scripting your browser to perform a number of tasks and then queries the Document Object Model (DOM) of the browser window. This can be easily done using F#.

Let's start by including Selenium WebDriver into our project:

We'll start by writing a simple test that will verify GitHub as the last link in the main navigation on my blog:

open OpenQA.Selenium

open OpenQA.Selenium.Chrome

open NUnit.Framework

[<Test>]

let ``my blog should have last link in main navigation to github`` () =

// open browser

use browser = new ChromeDriver()

// navigate to page

browser.Navigate().GoToUrl("http://blog.mikaellundin.name")

// find navigation

let navigation = browser.FindElementByClassName("navigation")

// extract links

let links = navigation.FindElements(By.XPath("//li/a"))

// get last link

let lastLink = links.[links.Count - 1]

// should contain github in address

Assert.That(lastLink.GetAttribute("href"), Contains.Substring("github"))

Here, I am using ChromeDriver. In order to get this to work, you need to download an executable and put it in the current working directory for your test runner. The executable can be found at http://chromedriver.storage.googleapis.com/index.html.

This was very useful, but not very reusable. There should also be a Twitter link in my main navigation, and in order to test this, I would have to write the same code all over again. In order to avoid doing so, we can introduce the Page Object pattern.

This simply means that a class can represent each page type that we're testing and the class can have properties, which are things you will find on the page, and also methods, which represent things you can do on the page, such as logging in. An example of this is as follows:

// represents <a href="">text</a>

type Link = { Href : string; Text : string }

// helper method to create a link from IWebElement

type Link with

static member Create (element : IWebElement) =

{ Href = element.GetAttribute("href"); Text = element.Text }

// wrapping up list of links into a Navigation type

type Navigation = Link list

// represents the start page

type StartPage (browser : IWebDriver) =

let urlPart = "/"

let baseUrl = ConfigurationManager.AppSettings.["BaseUrl"]

let navigationClassName = "navigation"

let navigationLinkXPath = "//li/a"

// move the browser session to the start page

member this.NavigateTo () =

browser.Navigate().GoToUrl(baseUrl + urlPart)

// the main navigation on the page

member this.Navigation : Navigation =

let navigationElement = browser.FindElement(By.ClassName(navigationClassName))

let linkElements = navigationElement.FindElements(By.XPath(navigationLinkXPath))

[ for linkElement in linkElements -> Link.Create(linkElement) ]

By extracting the details of a page into a type, we can reuse this in other tests running on the same page. When something changes in the HTML structure of that page, we only need to fix it in the type part. The tests that use this type stay the same.

Another great improvement made is related to the readability of the tests:

[<Test>]

let ``my blog should have a link to twitter in main navigation`` () =

// open browser

use browser = new ChromeDriver()

// create a start page

let startPage = new StartPage(browser)

// navigate to page

startPage.NavigateTo()

// find navigation

let navigation = startPage.Navigation

// has link to twitter

let hasLinkToTwitter = navigation |> List.exists (fun link -> link.Href.Contains("twitter"))

// assert

Assert.True(hasLinkToTwitter)

The beauty here is that the test doesn't know anything about the internals of the page structure. It only knows what it should expect of the page: that there is a navigation and that navigation contains a link. Then, it asserts that one of those links is indeed directed to Twitter.

As you may have noticed, I didn't output the URL to the page in the code this time around. Instead, I chose to put it in the appSettings file. This is important in order to run the same test on different environments. By changing the appSettings value, I can run the same test suite on my test, stage and production environments without breaking a sweat. Web browser-driven tests are also very useful when websites switch addresses, such as from HTTP to HTTPS.

The problem with web browser-driven tests is that those tests are really slow. Compared to a unit test that should execute in 1 ms, these tests are 5,000 times slower. This quickly becomes a problem because no developer will wait 50 seconds before checking in, as they are likely running 10 web tests at the same time.

I was running a project one time where we had quite a complex payment flow that we covered with web browser-driven tests. We had about 600 tests with a mean execution time of 40 seconds due to the complexity. There were simply a lot of things the browser would have to do before executing the actual test and asserting the result. If you do the math, you end up with between 5 and 6 hours to run the whole test suite, so we did this in the night. When we came back morning after, we received the report indicating whether we had broken something the previous day. This works fine in theory.

There were other things going on at night, however. There was a complete database backup job, that would make the whole site unresponsive for 20 minutes. This would cause a random set of tests to fail every night, and we would never know which ones. Once we got around this problem, we still had random false positives in the test run because of conflicts with night-based jobs.

The end result was that the test suite was green for 2 days out of the whole year. The effect this has on a team is that they simply don't look at the test report because it's always red. A red test run, at least in this case, however, doesn't mean the functionality is broken; it just means the test wasn't a success.

All this came from having very slow tests at a very large scale. Web-browser-based testing is hard to run in parallel because of session cookies. Once you start going down the path of optimizing your web browser-based tests, you will spend more time than it took to write them.

The following sections will propose what we can do about this.

PhantomJS

PhantomJS is a headless web browser, meaning it doesn't present pages in a graphical manner. Instead, it provides an API which you can use to control the browser and query its contents.

This is really useful for functional testing purposes. With this, you can easily test your web frontends without the need of opening a browser window. The browser window is a major side effect of your tests that can cause all kinds of trouble:

· The test suite needs to interact with the desktop, which is not always appropriate, for example, when you're running your tests on a build server. There are solutions for this, though.

· The test can fail at any given time because something else grabbed the focus of the browser window.

· The test can fail because something is trying to interact with the browser window, such as an installed add-on.

These are some of the headaches of browser testing, along with trusted zones and certificate errors. Some of these can be avoided by running a headless browser, such as PhantomJS, instead. This browser is based on WebKit, and so is closest to Safari and Chrome in its implementation.

Of course, this does not apply if what you're testing is cross-browser compatibility.

To enable this, we can use Selenium, which we're already familiar with. Before you can do this, you need to download the Windows package of PhantomJS from http://phantomjs.org/, or just download it using the NuGet Package Manager:

Now, we can write a simple test like this to make sure it works:

open OpenQA.Selenium

open OpenQA.Selenium.Internal

open OpenQA.Selenium.PhantomJS

open NUnit.Framework

open FsUnit

[<Test>]

let ``homepage should have 'Mikael Lundin' in the title`` () =

// arrange

use browser = new PhantomJSDriver()

// act

browser.Navigate().GoToUrl("http://mikaellundin.name")

// assert

browser.Title |> should contain "Mikael Lundin"

The rest consists of just following the standard Selenium implementation practices.

Sadly, it is still slow, so it doesn't remove all the pains of web browser-based testing; however, it does help with some of the more practical problems listed earlier. If we really need to reduce the time it takes to execute the test suite, we need to step away from the web browser for a while and look at our alternatives.

Canopy

There is an F# framework called Canopy that provides a frictionless functional layer on top of Selenium. The goal of this web testing framework is to provide a stable API that is quick to learn and is clean and concise.

Canopy tests run in a console application. Most testing frameworks have their tests built in a library, but with Canopy, you can build and execute your tests as an executable.

Make sure that in the properties of your test project, the framework version is > 4.0, and the output type is the console application:

All you have to do to get started is include Canopy in your testing project from the NuGet Package Manager window:

Now, you can begin to write your first test:

open canopy

open runner

open System

[<EntryPoint>]

let main argv =

//start an instance of the firefox browser

start firefox

// grouping for tests

context "Blog start page"

// define a test

"blog title should contain Mikael Lundin" &&& fun _ ->

// navigate

url "http://blog.mikaellundin.name"

// assert

contains "MIKAEL LUNDIN" (read "#blog-title")

// define a test

"number of post excerpts should be 5" &&& fun _ ->

// navigate

url "http://blog.mikaellundin.name"

// assert

count ".post-excerpt" 5

//run all tests

run()

// close browser window

quit()

// return an integer exit code

0

When running the executable, it will open a web browser, execute the tests, and then close the browser upon executing the quit() function. The output in the console window will be as follows:

context: Blog start page

Test: blog title should contain Mikael Lundin

Passed

Test: number of post excerpts should be 5

Passed

0 minutes 1 seconds to execute

2 passed

0 failed

The syntax that Canopy brings to web testing matches functional programming very neatly, but it does simplify the problem a bit too much and doesn't provide an API to support more advanced assertions that you would be able to perform with direct access to Selenium. Canopy may lower the bar for writing tests, but depending on what's being tested, using Selenium directly might be a better choice.

For full reference of Canopy, please visit its project page on GitHub at http://lefthandedgoat.github.io/canopy/.

CSQuery

In order to speed things up, the only option we have is to simplify the process, and we can do this by cutting out the web browser from the equation, as it is simply too slow.

In some cases where we want to test content and not interact, a browser is just in the way and we could settle with an HttpClient class. The messy part comes when you try to assert on the result, and this is where CSQuery comes into the picture.

CSQuery is a port of jQuery to C#, but of course, it's usable in all .NET languages. We can use it to run assertions on the DOM tree of the page we've downloaded with the HttpClient class.

Let's start by including CSQuery into our project:

Now, we can write a test that will download a page from a URL and assert on the page's information:

open System.Net

open CsQuery

open NUnit.Framework

open FsUnit

[<Test>]

let ``there should be at least 5 blog posts pushed from start page`` () =

// open new web client

use client = new WebClient()

// download the html we're testing

let html = client.DownloadString("http://blog.mikaellundin.name")

// parse the DOM

let dom = CQ(html)

// query the DOM for the blog posts

let posts = dom.[".post-excerpt"]

// assert that number of posts > 4

posts.Length |> should be (greaterThan 4)

This code does not offer much more ceremony than Selenium. We're using standard .NET functionality to get the HTML, and then we use CSQuery to parse the HTML.

This test runs in 500 ms, which makes it one-sixth of what the browser tests would take; however, it is still very slow because of the network traffic. If you're able to run your testing target locally and test against it, I'm sure you could get down to tens of milliseconds without a blink.

This is clearly very unsuitable for any scenario in which you interact with the web page and want to test that interaction. Let's say you log in and perform a number of tasks as an authenticated user; this is something you wouldn't want to use CSQuery for.

Another obvious thing is that scripts are not run on the page. If your tests rely on JavaScript, then CSQuery is completely useless to you. This will just download the HTML as is and assert on it.

The perfect scenario for CSQuery is when you want to test that the content has rendered correctly on the page. This is where you don't need a full-fledged browser to render the page, but could be satisfied with querying the resultant HTML.

Regression testing

Any test that doesn't bring value is a waste and should be deleted, because it still comes with the cost of maintenance. The hard part is to discover which tests are valueless and which are invaluable.

As a software tester, it is easy to draw the conclusion that every test provides regression, where regression means that it validates no new bugs appear in the functionality we covered with tests. This might be true in some cases, but it is wrong to make this assumption when it wasn't the purpose of the test. Unit and integration testing may provide some regression testing, but this comes as a side effect and not as its primary focus. In unit testing, the purpose is design, and in integration testing, the purpose is to validate how parts of the system fit together. These purposes have very little to do with regression.

When it comes to functional testing, regression is another matter. In a functional test, I claim to expect some functionality of the system from a black box perspective. We do not care how the system comes to the expected output, only that it does. When the implementation of the system changes, the unit tests will need to be refactored and the integration tests will fail, but the functional test will stay the same as long as the system stays true to its purpose. In this sense, it is a perfect regression test. Stating what we know to be true about a system is the only regression test that can be trusted.

The most common reason why the existing functionality stops working is that there is a change or fix made to the code that has a side effect the author didn't think about. Most often with these fixes, its not the original author of the code that does the fix. As such, this new developer might not understand the functionality completely, and in this regard, he or she fails to fulfill all its requirements. This is why there are regression tests in place, in order to ensure what we know about the function remains true after the change.

A good practice when dealing with changes in an existing code base is to start by expressing that change in a test that will turn up red until the change is implemented. This way, you're not only covered on old functionality, but also on any subsequent changes. This gives you a good regression test suite, as you only test on requirements that have been requested by the product owner and not on things you might believe will break, because this is often untrue. Before checking out the next chapter, there is a more advanced topic on test automation found online, Chapter 11, Property-Based Testing. If you thirst for more functional testing code, I suggest you head there before continuing with the rest of the book.

Summary

In the end, we want to have a fully featured test suite that describes how the system works. This specification will be the best documentation available for your system, as it will get verified every time a developer commits code to the solution. It will provide a layer of protection indicating that the functionality stays the same throughout the application life cycle.

The main challenge with functional testing will always be to create a test suite that runs fast enough and provides predictable results so we can trust the process. Functional tests are the hardest to write in plenty, and they still make a fast-running and stable test suite. We need to manage this in order to stay sane and keep providing value.