Writing Postfix unit tests


Overview

This document covers Ptest, a simple unit test framework that was introduced with Postfix version 3.8. It is modeled after Go tests, with primitives such as ptest_error() and ptest_fatal() that report test failures, and PTEST_RUN() that supports subtests.

Ptest is light-weight compared to more powerful frameworks such as Gtest, but it avoids the need for adding a large Postfix dependency (a dependency that would not affect Postfix distributors, but developers only).

Simple example

Simple tests exercise one function under test, one scenario at a time. Each scenario calls the function under test with good or bad inputs, and verifies that the function behaves as expected. The code in Postfix mymalloc_test.c file is a good example.

After some #include statements, the file goes like this:

 27 typedef struct PTEST_CASE {
 28     const char *testname;               /* Human-readable description */
 29     void    (*action) (PTEST_CTX *, const struct PTEST_CASE *);
 30 } PTEST_CASE;
 31 
 32 /* Test functions. */
 33 
 34 static void test_mymalloc_normal(PTEST_CTX *t, const PTEST_CASE *tp)
 35 {
 36     void   *ptr;
 37 
 38     ptr = mymalloc(100);
 39     myfree(ptr);
 40 }
 41 
 42 static void test_mymalloc_panic_too_small(PTEST_CTX *t, const PTEST_CASE *tp)
 43 {
 44     expect_ptest_log_event(t, "panic: mymalloc: requested length 0");
 45     (void) mymalloc(0);
 46     ptest_fatal(t, "mymalloc(0) returned");
 47 }
...     // Test functions for myrealloc(), mystrdup(), mymemdup().
260
261 static const PTEST_CASE ptestcases[] = {
262     {"mymalloc + myfree normal case", test_mymalloc_normal,
263     },
264     {"mymalloc panic for too small request", test_mymalloc_panic_too_small,
265     },
...     // Test cases for myrealloc(), mystrdup(), mymemdup().
306 };
307 
308 #include <ptest_main.h>

To run the test:

$ make test_mymalloc
... compiler output...
LD_LIBRARY_PATH=/path/to/postfix-source/lib ./mymalloc_test
RUN  mymalloc + myfree normal case
PASS mymalloc + myfree normal case
RUN  mymalloc panic for too small request
PASS mymalloc panic for too small request
... results for myrealloc(), mystrdup(), mymemdup()...
mymalloc_test: PASS: 22, SKIP: 0, FAIL: 0

This simple example already shows several key features of the ptest framework.

Testing one function with TEST_CASE data

Often, we want to test a module that contains only one function. In that case we can store all the test inputs and expected results in the PTEST_CASE structure.

The examples below are taken from the dict_union_test.c file which test the unionmap implementation in the file. dict_union.c.

Background: a unionmap creates a union of tables. For example, the lookup table "unionmap:{inline:{foo=one},inline:{foo=two}}" will return ("one, two", DICT_STAT_SUCCESS) when queried with foo, and will return (NOTFOUND, DICT_STAT_SUCCESS) otherwise.

First, we present the TEST_CASE structure with additional fields for inputs and expected results.

 29 #define MAX_PROBE       5
 30 
 31 struct probe {
 32     const char *query;
 33     const char *want_value;
 34     int     want_error;
 35 };
 36 
 37 typedef struct PTEST_CASE {
 38     const char *testname;
 39     void    (*action) (PTEST_CTX *, const struct PTEST_CASE *);
 40     const char *type_name;
 41     const struct probe probes[MAX_PROBE];
 42 } PTEST_CASE;

In the PTEST_CASE structure above:

Next we show the test data. Every test calls the same test_dict_union() function with a different unionmap configuration and with a list of queries with expected results. The implementation of that function follows after the test data.

 78 static const PTEST_CASE ptestcases[] = {
 79     {
 80          /* testname */ "successful lookup: static map + inline map",
 81          /* action */ test_dict_union,
 82          /* type_name */ "unionmap:{static:one,inline:{foo=two}}",
 83          /* probes */ {
 84             {"foo", "one,two", DICT_STAT_SUCCESS},
 85             {"bar", "one", DICT_STAT_SUCCESS},
 86         },
 87     }, {
 88          /* testname */ "error propagation: static map + fail map",
 89          /* action */ test_dict_union,
 90          /* type_name */ "unionmap:{static:one,fail:fail}",
 91          /* probes */ {
 92             {"foo", 0, DICT_STAT_ERROR},
 93         },
...
102 };
103 
104 #include <ptest_main.h>

Finally, here is the test_dict_union() function that tests the unionmap implementation with a given configuration and test queries.

 44 #define STR_OR_NULL(s)  ((s) ? (s) : "null")
 45 
 46 static void test_dict_union(PTEST_CTX *t, const struct PTEST_CASE *tp)
 47 {
 48     DICT   *dict;
 49     const struct probe *pp;
 50     const char *got_value;
 51     int     got_error;
 52 
 53     if ((dict = dict_open(tp->type_name, O_RDONLY, 0)) == 0)
 54         ptest_fatal(t, "dict_open(\"%s\", O_RDONLY, 0) failed: %m",
 55                     tp->type_name);
 56     for (pp = tp->probes; pp < tp->probes + MAX_PROBE && pp->query != 0; pp++) {
 57         got_value = dict_get(dict, pp->query);
 58         got_error = dict->error;
 59         if (got_value == 0 && pp->want_value == 0)
 60             continue;
 61         if (got_value == 0 || pp->want_value == 0) {
 62             ptest_error(t, "dict_get(dict, \"%s\"): got '%s', want '%s'",
 63                         pp->query, STR_OR_NULL(got_value),
 64                         STR_OR_NULL(pp->want_value));
 65             break;
 66         }
 67         if (strcmp(got_value, pp->want_value) != 0) {
 68             ptest_error(t, "dict_get(dict, \"%s\"): got '%s', want '%s'",
 69                         pp->query, got_value, pp->want_value);
 70         }
 71         if (got_error != pp->want_error)
 72             ptest_error(t, "dict_get(dict,\"%s\") error: got %d, want %d",
 73                         pp->query, got_error, pp->want_error);
 74     }
 75     dict_free(dict);
 76 }

A test run looks like this:

$ make test_dict_union
...compiler output...
LD_LIBRARY_PATH=/path/to/postfix-source/lib ./dict_union_test
RUN  successful lookup: static map + inline map
PASS successful lookup: static map + inline map
RUN  error propagation: static map + fail map
PASS error propagation: static map + fail map
...
dict_union_test: PASS: 3, SKIP: 0, FAIL: 0

Testing functions with subtests

Sometimes it is not convenient to store test data in a PTEST_CASE structure. This can happen when converting an existing test into Ptest, or when the module under test contains multiple functions that need different kinds of test data. The solution is to create a _test.c file with the structure shown below. The example is based on code in map_search_test.c that was converted from an existing test into Ptest.

See the file map_search_test.c for a complete example.

This is what a test run looks like:

$ make test_map_search
...compiler output...
LD_LIBRARY_PATH=/path/to/postfix-source/lib  ./map_search_test
RUN  test_map_search
RUN  test_map_search/test 0
PASS test_map_search/test 0
....
PASS test_map_search
map_search_test: PASS: 13, SKIP: 0, FAIL: 0

This shows that the subtest name is appended to the parent test name, formatted as parent-name/child-name.

Suggestions for writing tests

Ptest is loosely inspired on Go test, especially its top-level test functions and its methods T.run(), T.error() and T.fatal().

Suggestions for test style may look familiar to Go programmers:

Other suggestions:

Ptest API reference

Managing test errors

As one might expect, Ptest has support to flag unexpected test results as errors.

void ptest_error(PTEST_CTX *t, const char *format, ...)
Called from inside a test to report an unexpected test result, and to flag the test as failed without terminating the test. This call can be ignored with expect_ptest_error().

void ptest_fatal(PTEST_CTX *t, const char *format, ...)
Called from inside a test to report an unexpected test result, to flag the test as failed, and to terminate the test. This call cannot be ignored with expect_ptest_error().

For convenience, Ptest has can also report non-error information.

void ptest_info(PTEST_CTX *t, const char *format, ...)
Called from inside a test to report a non-error condition without terminating the test. This call cannot be ignored with expect_ptest_error().

Finally, Ptest has support to test ptest_error() itself, to verify that an intentional error is reported as expected.

void expect_ptest_error(PTEST_CTX *t, const char *text)
Called from inside a test to expect exactly one ptest_error() call with the specified text, and to ignore that ptest_error() call (i.e. don't flag the test as failed). To ignore multiple calls, call expect_ptest_error() multiple times. A test is flagged as failed when an expected error is not reported (and of course when an error is reported that is not expected with expect_ptest_error()).

Managing log events

Ptest integrates with Postfix msg(3) logging.

Ptest provides the following API to manage log events:

void expect_ptest_log_event(PTEST_CTX *t, const char *text)
Called from inside a test to expect exactly one msg(3) call with the specified text. To expect multiple events, call expect_ptest_log_event() multiple times. A test is flagged as failed when expected text is not logged, or when text is logged that is not expected with expect_ptest_log_event().

Managing test execution

Ptest has a number of primitives that control test execution.

void PTEST_RUN(PTEST_CTX *t, const char *test_name, { code in braces })
Called from inside a test to run the { code in braces } in it own subtest environment. In the test progress report, the subtest name is appended to the parent test name, formatted as parent-name/child-name.

NOTE: because PTEST_RUN() is a macro, the { code in braces } must not contain a return statement; use ptest_return() instead. It is OK for { code in braces } to call a function that uses return.

NORETURN ptest_skip(PTEST_CTX *t)
Called from inside a test to flag a test as skipped, and to terminate the test without terminating the process. Use this to disable tests that are not applicable for a specific system type or build configuration.

NORETURN ptest_return(PTEST_CTX *t)
Called from inside a test to terminate the test without terminating the process.

void ptest_defer(PTEST_CTX *t, void (*defer_fn)(void *), void *defer_ctx)
Called once from inside a test, to call defer_fn(defer_ctx) after the test completes. This is typically used to eliminate a resource leak in tests that terminate the test early.

NOTE: The deferred function is designed to run outside a test, and therefore it must not call Ptest functions.