Chapter 14: Testing

“Program testing can be used to show the presence of bugs, but never to show their absence!” ~ Edsger W. Dijkstra

“Beware of bugs in the above code; I have only proved it correct, not tried it.” ~ Donald Knuth

14.1 Testing

Although Haskell emphasizes program correctness by construction[1], no amount of inductive reasoning is as convincing to stakeholders as exercising your code for its intended usage.

[1]: To get a sense of what I mean, see “Program = Proof”, by Samuel Mimram.

Worse yet, without constant communication with the customer and empirical testing, you may find that you’ve created a perfectly consistent formal model of something that doesn’t perform tasks the customer intended you to automate.

Automated tests are a limited form of empirical testing – executable sanity checks – and an indispensable tool for working programmers.

Really, we need all three – constant communication with the customer, empirical testing, and proofs of correctness (using the type system, model checkers, etc).

This chapter of the book is pretty narrowly focused on covering how to use Hspec and QuickCheck to write tests in Haskell. There is also a long example program at the end of the chapter that demonstrates using these libraries to test a Morse code translator.

Since I’m new to testing, I’ve tried to include exposition on basic concepts to fill in the gaps. Mostly this takes the form of paraphrases from different articles on the web.

I’m assuming one day I’ll look back at these notes and shake my head in disapproval at how wrong I’ve gotten everything; But for now I view this extra commentary as a useful dialog with myself to explore the topic of testing.

If you found these notes from a web search, don’t take them too seriously. I’m only a beginner.

14.2 A quick tour of testing for the uninitiated

Whenever you load your code into GHCi to play with a function you wrote, you’re testing your code; you’re just doing it manually.

In general, automated tests allow you to state an expectation and then verify that the result of an operation meets that expectation. Just like experimenting in the REPL, tests allow you to verify that you code will do what you want when executed.

There are multiple categories of automated tests, categorized roughly by what they are intended to test.

If you look up “software testing basics” with a search engine, you may find yourself looking at an overwhelming listicle of over a thousand terms for arbitrary categorizations of kinds of tests or testing methodologies.

Essentially, though, tests are just code, and the categorization doesn’t matter. Simply keep in mind the end goal – you want to prove that your software works.

To do that you’ll want to test that the intent of the program matches the tasks it can perform. You should also test that the entire program works. Along the way, you’ll probably have to test individual components of your program to make sure they work, too.

The last of these is called unit testing. Unit tests exercise the smallest atomic units of software independently of one another to make sure they work in isolation. In a unit test, usually you write the function applied to a particular input, and test that it produces a particular expected output. Hspec is the tool we’ll use in this chapter to write unit tests.

As a beginner writing his own code, this is the first type of test I’ve encountered. Perhaps if I were to contribute to someone else’s project, I’d have encounter an end-to-end test first.

Another useful tool that Haskell programmers use is called property testing. The general idea is to generate random inputs that are checked against a test function, known as a property, to see if it holds.

Here is a sample of a property test, to show what I mean:

import Test.QuickCheck

prop_reverseReverse :: Eq a => [a] -> Bool
prop_reverseReverse xs = reverse (reverse xs) == xs

main = quickCheck prop_reverseReverse

When main is run, prop_reverseReverse will be fed random inputs of type Eq a => [a] to see if our condition still returns True.

If unit testing is essentially automating manual tests, then property testing is automating unit tests.

QuickCheck is the package that provides property testing in Haskell. It relies on your functions type signature to know what kinds of input to generate. The default setting is for 100 inputs to be generated, giving you 100 results.

QuickCheck is cleverly written to be as thorough as possible and will usually check the most common edge cases (for example empty lists and the maxBound and minBound of the types in question).

If the function being tested fails any of these tests, we know the function doesn’t have the specified property. On the other hand, you can’t be positive that it will never fail because the data are randomly generated.

Property testing is useful for getting a strong indication that you’ve met the minimum requirements to satisfy laws, such as the laws of monads or basic associativity.

Other testing tools

Hspec and QuickCheck are only the tip of the testing iceberg. Check out SmallCheck, a library for testing every possible input of a given type, and Tasty, a unit testing framework that integrates multiple other tools. Diehl has a brief description of some of these here. Another intersting library to look at is hedgehog.

14.3 Conventional testing

First, let’s set up a project that we’ll put our tests into.

$ mkdir addition && cd addition

$ cat > addition.cabal << EOF
name: addition
version: 0.1.0.0
author: Chicken Little
maintainer: sky@isfalling.org
category: Text
build-type: Simple
cabal-version: >=1.10

library
  exposed-modules: Addition
  ghc-options: -Wall -fwarn-tabs
  build-depends: base >= 4.7 && <5, hspec
  hs-source-dirs: .
  default-language: Haskell2010

EOF

$ stack init

$ stack build

Now that we’ve made the project skeleton, and stack can build it, we’ll enter ghci, and then test that the functions from Addition are in scope.

$ stack ghci

Configuring GHCi with the following packages: addition
GHCi, version 8.10.3: https://www.haskell.org/ghc/ :? for help
Loaded GHCi configuration from /home/chris/.ghci
[1 of 1] Compiling Addition   ( Addition.hs, interpreted )
Ok, one module loaded.
Loaded GHCi configuration from
/tmp/haskell-stack-ghci/1506c361/ ghci-script

·∾ sayHello
hello!

14.3.1 Truth according to Hspec

Let’s experiment with writing unit tests with Hspec.

To do so, we’ll need to make the package available to our project. Add the hspec package to build-depends in your cabal file.

Now we must bring it into scope. Import the module Test.Hspec in Addition.hs so we can use it.

14.3.2 Our first Hspec test

Here is a simple example of a unit test using Hspec:

module Addition where
import Test.Hspec
import Test.QuickCheck



dividedBy :: Integral a => a -> a -> (a, a)
dividedBy num denom =
  go 0 num denom
  where
    go count n d =
      if n < d
      then (count,n)
      else go (count+1) (n-d) d



-- from page 551
-- Play with these in the repl by using sample on them.
genBool :: Gen Bool
genBool = choose (False,True)

genBool' :: Gen Bool
genBool' = elements [False,True]

genOrdering :: Gen Ordering
genOrdering = elements [LT,EQ,GT]

genChar :: Gen Char
genChar = elements ['a'..'z']



-- More complex examples.
genTuple :: (Arbitrary a, Arbitrary b) => Gen (a,b)
genTuple = do
  a <- arbitrary
  b <- arbitrary
  return (a,b)

genThreeple :: (Arbitrary a, Arbitrary b, Arbitrary c) => Gen (a,b,c)
genThreeple = do
  a <- arbitrary
  b <- arbitrary
  c <- arbitrary
  return (a,b,c)



-- page 553
genEither :: (Arbitrary a, Arbitrary b) => Gen (Either a b)
genEither = do
  a <- arbitrary
  b <- arbitrary
  elements [Left a, Right b]

genMaybe :: Arbitrary a => Gen (Maybe a)
genMaybe = do
  a <- arbitrary
  elements [Nothing, Just a]

genMaybe' :: Arbitrary a => Gen (Maybe a)
genMaybe' = do
  a <- arbitrary
  frequency [ (1, return Nothing)
            , (3, return (Just a))
            ]
-- ·∾ :type frequency
-- frequency :: [(Int, Gen a)] -> Gen a
-- ·∾ :doc frequency
-- Chooses one of the given generators, with a weighted
-- random distribution. The input list must be non-empty.



main :: IO ()
main = hspec $ do

  describe "Addition" $ do
    it "1 + 1 is greater than 1" $ do
      (1 + 1) > 1 `shouldBe` True
    it "x+1 is always greater than x" $ do
      property $ \x -> x + 1 > (x :: Int)

  describe "Division" $ do
    it "15 divided by 3 is  5" $ do
      dividedBy 15 3 `shouldBe` (5, 0)
    it "22 divided by 5 is 4 remainder 2" $ do
      dividedBy 22 5 `shouldBe` (4, 2)

14.3.3 Intermission: Short Exercise

In the chapter exercises at the end of recursion, you were given this exercise:

Write a function that multiplies two numbers using recursive summation. The type should be (Eq a, Num a) => a -> a -> a although, depending on how you do it, you might also consider adding an Ord constraint.

If you still have your answer, great! If not, rewrite it and then write hspec tests for it.

Ok partner, I quickly rewrote mult. Here’s what it looks like:

module Lib where

mult x y =
  case signum y of
    0    -> 0
    1    -> incrementBy 1 x
    (-1) -> negate (incrementBy 1 x)
  where
    incrementBy counter n =
      if counter /= abs y
      then incrementBy (counter+1) (n+x)
      else n

…and here is the associated test suite:

import Lib
import Test.Hspec
import Test.QuickCheck

main :: IO ()
main = hspec $ do
  describe "mult" $ do
    context "associativity" $ do
      it "x `mult` y always equals y `mult` x" $ do
        property (\x y ->
          (x :: Int) `mult` (y :: Int) == y `mult` x)
    context "negative numbers" $ do
      it "(-1) `mult` 0 returns 0" $ do
        (-1) `mult` 0 `shouldBe` 0
      it "(-7) 2 -> (-14)" $ do
        mult (-7) 2 `shouldBe` (-14)
      it "10 (-2) -> (-20)" $ do
        mult 10 (-2) `shouldBe` (-20)
      it "(-10) (-2) -> 20" $ do
        mult (-10) (-2) `shouldBe` 20
    context "positive numbers" $ do
      it "5 5 -> 25" $ do
        mult 5 5 `shouldBe` 25
    context "multiplication by zero" $ do
      it "8 0 -> 0" $ do
        mult 8 0 `shouldBe` 0
      it "0 8 -> 0" $ do
        mult 0 8 `shouldBe` 0

You can run this by navigating to exercises/ 14.3.3_-_intermission_short_exercise.rst.d/mult and running stack test.

The above examples demonstrate the basics of writing individual tests to test particular values. If you’d like to see a more developed example, you could refer to Chris’s library, Bloodhound.

14.4 Enter QuickCheck

There are two ways to run a QuickCheck test described in this book.

The first is by using Hspec in combination with QuickCheck, like this:

import Test.Hspec
import Test.QuickCheck

main :: IO ()
main = hspec $ do
  describe "Addition" $ do
    it "x + 1 is always greater than x" $ do
      property $ \x -> x + 1 > (x :: Int)

Another way to run QuickCheck tests is to use the facilities provided by the Test.QuickCheck module, like this:

import Test.QuickCheck

preprocess s = filter isAlpha (map toLower s)

isPalindrome :: String -> Bool
isPalindrome s = (preprocess s) == reverse (preprocess s)

prop_punctuationInvariant text =
  preprocess text == preprocess noPuncText
  where noPuncText = filter (not . isPunctuation) text

main = quickCheck prop_punctuationInvariant

QuickCheck also provides something called conditional properties. Using the (==>) function we can filter out inputs from being tested based on some precondition.

This has the general form condition ==> property.

Here is a simple example of its use:

import Test.QuickCheck

qsort :: [Int] -> [Int]
qsort []     = []
qsort (x:xs) = qsort lhs ++ [x] ++ qsort rhs
    where lhs = filter  (< x) xs
          rhs = filter (>= x) xs

prop_maximum ::  [Int] -> Property
prop_maximum xs = not (null xs) ==>
                  last (qsort xs) == maximum xs


main :: IO ()
main = quickCheck prop_maximum

In this function, only values of type [Int] that are not empty lists (null) are permitted as inputs for our test.

Properties may take the general form:

forAll generator $ \pattern -> property

For example:

prop_Insert2 x = forAll orderedList $ \xs -> ordered (insert x xs)
  where types = x::Int

The first argument of forAll is a test data generator; by supplying a custom generator, instead of using the default generator for that type, it is possible to control the distribution of test data.

14.4.1 Arbitrary instances

The tricky part of QuickCheck is generating the input values to test on. All types that QuickCheck can automatically test must be an instance of the type class Arbitrary.

The bad news is that only a few base types are instances of Arbitrary. The good news is that you can install package that greatly extends the types covered by QuickCheck, named quickcheck-instances.

If you want to write an instance of Arbitrary for your own custom type, the chapter touches on that. But it’s way over my head to be honest.

One thing that may be useful is knowing how to print sample values of some type that QuickCheck can generate.

We can use the sample function for this, in combination with an overloaded expression from the Arbitrary type class named arbitrary:

·∾ sample (arbitrary :: Gen Int)
0
2
-4
5
2
3
-10
-1
3
12
-2

·∾ sample (arbitrary :: Gen Double)
0.0
0.6341225463105274
-0.5399666722390497
3.6986851136506376
4.927328536143319
-0.34216302388027836
4.401389073625471
5.706335581327833
14.466727278626447
-0.5275031627254437
-8.811337993125159

As you can see, the sample function has produced some random sample data of the types we’ve specified (Gen Int and Gen Double respectively). The sample function is what has introduced randomness here, arbitrary by itself is not random.

Knowing what you do about referential transparency, you may be wondering how these functions produce random data. After all, the definition of a pure function is that it always produces the same results given the same input.

The answer is apparent if you examine the type signature of sample:

·∾ :type sample
sample :: Show a => Gen a -> IO ()

It turns out, sample is not a pure function, but an IO action. It needs to be, so it can ingest a source of randomness.

In this section, a few other means of generating sample data are demonstrated. This includes the elements, frequency, and choose functions.

A short summary:

·∾ :type sample
sample :: Show a => Gen a -> IO ()

·∾ :type sample'
sample' :: Gen a -> IO [a]

·∾ :type choose
choose :: Random a => (a, a) -> Gen a

·∾ :type elements
elements :: [a] -> Gen a

·∾ :type frequency
frequency :: [(Int, Gen a)] -> Gen a

At this point, it’s not clear to me how I may use them in my own programs, or why it’s being discussed.

14.5 Morse code

In the interest of playing with testing, we’ll work through an example project where we translate text to and from Morse code.

Peruse the projects/morse directory to view what I’ve copied from the book there.

14.6 Arbitrary instances

14.6.1 Babby’s First Arbitrary

·∾ import Test.QuickCheck
·∾ data Trivial = Trivial deriving (Eq, Show)
·∾ trivialGen = return Trivial :: Gen Trivial
·∾ instance Arbitrary Trivial where { arbitrary = trivialGen }
·∾ :{
 ⋮ main :: IO ()
 ⋮ main = sample trivialGen
 ⋮ :}
·∾
·∾ main
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
Trivial
·∾

14.6.2 Identity Crisis

·∾ data Identity a = Identity a deriving (Eq, Show)
·∾ :{
 ⋮ identityGen :: Arbitrary a => Gen (Identity a)
 ⋮ identityGen = do
 ⋮   a <- arbitrary
 ⋮   return (Identity a)
 ⋮
 ⋮ instance Arbitrary a => Arbitrary (Identity a) where
 ⋮   arbitrary = identityGen
 ⋮
 ⋮ identityGenInt :: Gen (Identity Int)
 ⋮ identityGenInt = identityGen
 ⋮ :}
·∾
·∾ sample identityGenInt
Identity 0
Identity (-2)
Identity 4
Identity 2
Identity (-5)
Identity 5
Identity (-5)
Identity (-7)
Identity (-14)
Identity 8
Identity (-12)
·∾

14.7 Chapter Exercises

14.7.2 Using QuickCheck

Test some basic properties using QuickCheck.

1. For a function

   half x = x / 2
   

   This property should hold:

   halfIdentity = (*2) . half
   

2. import Data.List (sort)
   
   -- for any list you apply sort to this property should hold
   listOrdered :: (Ord a) => [a] -> Bool
   listOrdered xs =
     snd $ foldr go (Nothing, True) xs
     where go _ status@(_, False) = status
       go y (Nothing, t) = (Just y, t)
       go y (Just x, t) = (Just y, x >= y)
   

3. Now we’ll test the associative and commutative properties of addition

   plusAssociative x y z  =  x + (y + z) == (x + y) + z
   plusCommutative x y    =  x + y == y + x
   

   Keep in mind these properties won’t hold for types based on IEEE-754 floating point numbers, such as Float or Double.

4. Now do the same for multiplication.

5. We mentioned in one of the first chapters that there are some laws involving the relationship of quot and rem and div and mod. Write QuickCheck tests to prove them.

   -- quot rem
   (quot x y)*y + (rem x y) == x
   (div x y)*y + (mod x y) == x
   

6. Is (^) associative? Is it commutative? Use QuickCheck to see if the computer can contradict such an assertion.

7. Test that reversing a list twice is the same as the identity of the list:

   reverse . reverse == id
   

8. Write a property for the definition of ($).

   f $ a = f a
   f . g = \x -> f (g x)
   

9. See if these two functions are equal

   foldr (:) == (++)
   foldr (++) [] == concat
   

10. Hmm. Is that so?

    f n xs = length (take n xs) == n
    

11. Finally, this is a fun one. You may remember we had you compose read and show one time to complete a “round trip.” Well, now you can test that it works:

    f x = (read (show x)) == x