Property-based testing

Headline

Testing functions based on the expected properties with the help of data generators

Property-based testing is a testing technique in which we specify general properties that a program should satisfy, rather than enumerating individual example inputs and expected outputs. A property describes a law, invariant, round-trip behavior, algebraic identity, or relationship between operations; the testing tool (such as QuickCheck in the case of Haskell) then generates many test cases automatically and checks whether the property holds for all generated inputs. When a property fails, the tool typically reports a concrete counterexample and may shrink it to a simpler failing case, helping the developer understand the defect. This style of testing is especially useful for data structures, parsers and printers, encoders and decoders, APIs, and mathematical or algebraic code, because it encourages thinking in terms of specifications rather than isolated examples. It does not replace example-based testing, but complements it by exploring a much larger input space and by making assumptions about program behavior explicit.

Details

We describe property-based testing here with the help of QuickCheck-based illustrations.

Behavioral law

Assume we want to test a stack implementation. We would assume these algebraic equalities:

top (push x s) = x
pop (push x s) = s

We turn these expected laws into code like this:

topProperty :: Stack Int -> Int -> Bool
topProperty s x =
  top (push x s) = x
popProperty :: Stack Int -> Int -> Bool
popProperty s x =
  pop (push x s) = s

We can also use an abstract data type Property:

prop_topPush :: Int -> Stack Int -> Property
prop_topPush x s =
  top (push x s) === x
  
prop_popPush :: Int -> Stack Int -> Property
prop_popPush x s =
  toList (pop (push x s)) === toList s

Input-space exploration

Once we have a behavioral law, we need evidence. Property-based testing obtains this evidence by exploring an input space automatically. For ordinary Haskell types, QuickCheck already knows how to generate test data. Consider this property:

prop_reverseReverse :: [Int] -> Property
prop_reverseReverse xs =
  reverse (reverse xs) === xs

Here the input space is “lists of integers.” QuickCheck supplies empty lists, singleton lists, short lists, longer lists, positive and negative numbers, repeated values, and so on.

For user-defined or abstract types, we provide generation logic through the type class Arbitrary. For example:

instance Arbitrary a => Arbitrary (Stack a) where
  arbitrary = fromList <$> arbitrary

This says: generate a random list, then convert it into a stack.

The broader concept is input-space exploration: rather than hand-picking a few cases, we define the domain of possible tests and let the tool sample it systematically.

Test-data modeling

Generation is not merely a technical detail. It is test-data modeling: we decide what valid, representative, unusual, or boundary-like data should look like. For instance, given a 101companies-style model, we may introduce wrappers that are not part of the domain model itself, but part of the testing model; this would be necessary specifically, when we use type synonyms in the domain model or when we want several testing distributions for the same underlying type. For example:

newtype AnyCompany = AnyCompany Company
  deriving Show

newtype RankedCompany = RankedCompany Company
  deriving Show

newtype SmallCompany = SmallCompany Company
  deriving Show

newtype DeepCompany = DeepCompany Company
  deriving Show

newtype AnyDepartment = AnyDepartment Department
  deriving Show
  
instance Arbitrary AnyCompany where
  arbitrary = AnyCompany <$> genCompany

instance Arbitrary AnyDepartment where
  arbitrary = AnyDepartment <$> genDepartment
  
...

prop_ranking_accepts_well_ranked :: RankedCompany -> Property
prop_ranking_accepts_well_ranked (RankedCompany c) =
  ranking c === True

Typical combinators for describing the generation of data are these:

choose    :: Random a => (a, a) -> Gen a
chooseInt :: (Int, Int) -> Gen Int
elements  :: [a] -> Gen a
oneof     :: [Gen a] -> Gen a
frequency :: [(Int, Gen a)] -> Gen a
listOf    :: Gen a -> Gen [a]
listOf1   :: Gen a -> Gen [a]
vectorOf  :: Int -> Gen a -> Gen [a]

Here are examples of generators as they could be used in Arbitrary instances:

genEmployee :: Gen Employee
genEmployee = do
  name <- elements ["Alice", "Bob", "Carol", "Dave"]
  address <- elements ["A1", "A2", "A3"]
  salary <- choose (0, 100000)
  pure (name, address, salary)
  
genDepartment :: Gen Department
genDepartment = do
  name <- elements ["Research", "Sales", "HR", "IT"]
  manager <- genEmployee
  subunits <- listOf genSubunit
  pure (name, manager, subunits)

Complexity control

Generated data must remain finite, diverse, and computationally affordable. This is the role of complexity control. For non-recursive or list-based structures, QuickCheck’s built-in generators often handle size well enough. For recursive data, however, we need to manage size explicitly. Consider an expression language:

data Expr
  = Lit Int
  | Add Expr Expr
  | Neg Expr
  deriving (Eq, Show)

A naive generator may recurse forever:

instance Arbitrary Expr where
  arbitrary =
    oneof
      [ Lit <$> arbitrary
      , Add <$> arbitrary <*> arbitrary
      , Neg <$> arbitrary
      ]

The size-aware version uses sized:

sized :: (Int -> Gen a) -> Gen a

For example:

instance Arbitrary Expr where
  arbitrary = sized genExpr
  
genExpr :: Int -> Gen Expr
genExpr 0 =
  Lit <$> arbitrary

genExpr n =
  frequency
    [ (5, Lit <$> arbitrary)
    , (3, Add <$> genExpr (n `div` 2) <*> genExpr (n `div` 2))
    , (2, Neg <$> genExpr (n - 1))
    ]

The important point is that recursive calls consume the size budget:

genExpr (n `div` 2)
genExpr (n - 1)

Other useful API:

resize :: Int -> Gen a -> Gen a
scale  :: (Int -> Int) -> Gen a -> Gen a

For example:

smallExpr :: Gen Expr
smallExpr =
  resize 5 arbitrary
  
smaller :: Gen a -> Gen a
smaller =
  scale (`div` 2)

Validity conditions

Some properties are only meaningful for valid inputs. This is the concept of validity condition: a precondition that must hold before the behavior under test is defined or meaningful. For a simple stack ADT, top and pop are partial: they are only defined for non-empty stacks. One way to express this is implication:

prop_popDecreasesSize :: Stack Int -> Property
prop_popDecreasesSize s =
  not (isEmpty s) ==>
    size (pop s) === size s - 1

Here we use this API:

(==>) :: Testable prop => Bool -> prop -> Property

However, implication can discard many generated tests. Often it is better to generate valid data directly - without precondition:

newtype NonEmptyStack a = NonEmptyStack (Stack a)
  deriving Show

instance Arbitrary a => Arbitrary (NonEmptyStack a) where
  arbitrary = do
    x <- arbitrary
    s <- arbitrary
    pure (NonEmptyStack (push x s))

prop_popDecreasesSize_direct :: NonEmptyStack Int -> Property
prop_popDecreasesSize_direct (NonEmptyStack s) =
  size (pop s) === size s - 1

QuickCheck also provides standard modifiers:

Positive a
NonNegative a
NonZero a
NonEmptyList a
Small a
Large a

For example:

prop_safeDiv :: Int -> NonZero Int -> Property
prop_safeDiv x (NonZero y) =
  safeDiv x y === Just (x `div` y)

The broader message: property-based testing makes preconditions visible. It forces us to ask whether an operation is total, partial, or only meaningful over a constrained domain.

Observability and oracles

A property needs a way to decide whether observed behavior is correct. That decision mechanism is the oracle.

Sometimes the oracle is an algebraic law:

prop_reverseAppend :: [Int] -> [Int] -> Property
prop_reverseAppend xs ys =
  reverse (xs ++ ys) === reverse ys ++ reverse xs

Sometimes it is a round-trip:

prop_showReadRoundtrip :: Company -> Property
prop_showReadRoundtrip c =
  read (show c) === c

Sometimes it is an independent reference implementation:

prop_total_is_sum_of_salaries :: AnyCompany -> Property
prop_total_is_sum_of_salaries (AnyCompany c) =
  total c `approxEqProp` sum (allSalaries c)

For floating-point data, exact equality may be the wrong oracle. Instead, use approximate equality:

approxEq :: Float -> Float -> Bool
approxEq x y =
  abs (x - y) <= 1e-3 * max 1 (max (abs x) (abs y))

A property can then be written as:

approxEqProp :: Float -> Float -> Property
approxEqProp x y =
  counterexample msg $
    approxEq x y
  where
    msg =
      "left = " ++ show x ++
      ", right = " ++ show y

prop_total_concat_departments :: [Department] -> [Department] -> Property
prop_total_concat_departments xs ys =
  total ("C", xs ++ ys)
    `approxEqProp`
  (total ("C", xs) + total ("C", ys))

The relevant API is this:

counterexample :: Testable prop => String -> prop -> Property
conjoin        :: Testable prop => [prop] -> Property

Here is one more illustrative example with several observations:

prop_stack_observations :: Int -> Stack Int -> Property
prop_stack_observations x s =
  conjoin
    [ top (push x s) === x
    , size (push x s) === size s + 1
    , toList (pop (push x s)) === toList s
    ]

To summarize, properties depend on oracles, and oracles are design choices. A failure may expose a program defect, but it may also expose a flawed specification, a poor equality notion, or an unrealistic model.

Diagnostic feedback

Property-based testing is valuable not only because it finds failures, but because it gives useful diagnostic feedback. When a property fails, QuickCheck reports a counterexample. With shrinking, it tries to reduce the counterexample to a simpler failing case. We can improve feedback when using === amd Property instead of == and Bool.

For example, we can add custom messages:

prop_sizeToList :: Stack Int -> Property
prop_sizeToList s =
  counterexample ("stack observed as " ++ show (toList s)) $
    size s === length (toList s)

We can also inspect what kinds of tests were generated:

prop_stack_classified :: Stack Int -> Property
prop_stack_classified s =
  classify (isEmpty s) "empty" $
  classify (size s == 1) "singleton" $
  classify (size s > 10) "large" $
    size s === length (toList s)

We can also collect observed values:

prop_stack_collect_size :: Stack Int -> Property
prop_stack_collect_size s =
  collect (size s) $
    size s === length (toList s)

We can also require coverage:

prop_stack_coverage :: Stack Int -> Property
prop_stack_coverage s =
  cover 10 (isEmpty s) "empty stacks" $
  cover 30 (size s > 3) "non-trivial stacks" $
    size s === length (toList s)

The relevant API is this:

classify :: Testable prop => Bool -> String -> prop -> Property
collect  :: (Show a, Testable prop) => a -> prop -> Property
cover    :: Testable prop => Double -> Bool -> String -> prop -> Property
label    :: Testable prop => String -> prop -> Property

Here is we run tests with different settings:

quickCheckWith stdArgs { maxSuccess = 1000 } prop_stack_coverage

Here are useful runners:

quickCheck
verboseCheck
quickCheckWith
stdArgs

Metadata

◀

this

sameAs

https://en.wikipedia.org/wiki/Software_testing#Property_testing

◀

this

memberOf

Vocabulary:Software engineering

◀

this

memberOf

Vocabulary:Programming

◀

this

instanceOf

Testing

Backlinks

Script:Property-based testing

Ralf Lämmel edited this article at Sat, 20 Jun 2026 20:00:10 +0200

This work is licensed under a Creative Commons Attribution 4.0 International License.

Concept:

Property-based testing

Headline

Description

Details

Behavioral law

Input-space exploration

Test-data modeling

Complexity control

Validity conditions

Observability and oracles

Diagnostic feedback

Metadata

Backlinks

User contributions

User edits

Syntax for editing wiki

Syntax for editing wiki