Test ALL the Things – On improving the nixpkgs testing story by Profpatsch (NixCon 2017)

While there is pretty good support for testing NixOS based services inside VMs, the same doesn’t hold for large parts of nixpkgs, especially for testing packages. I will present different approaches to improve testing.

Over the last year I have been creating various testing environments, tried out different approaches and created tests for various parts of nixpkgs, notably package tests with very interesting properties.

There are two kinds of tests in nixpkgs right now:

/lib/tests: this is essentially a derivation that tests (some/most) library functions by checking their outputs.

Both kinds of tests live in a different part of the codebase than the implementations they test, and have to be run separately, and manually (by hand or by CI). This leads to easy desynchronization of test- and actual implementation—the tests bit-rot over time.

I therefore propose two maxims:

All testing code should be as close to the implementation code as possible.
Changing the implementation should not be possible without adjusting the testing code to make it pass.

Coming from this angle, I recognized that a unique property of nix evaluation—short runtime—makes it possible to let nix packages depend on their tests. This is accomplished by a `seq` for derivations, discovered with @aszlig.

I will present how drvSeq makes it possible to create package tests that are able to use the full power of the nix language for abstraction. It also does not cause unnecessary recompilation, like changing the checkPhase does right now.
Going further, I will discuss best practices for including these tests into already existing packages.

As well as packages, functions (and thereby whole language frameworks) can be tested as well, enabling a kind of testing that is automatically done to every usage of the function (i.e. every package in the framework). This could lead to interesting use cases.

While VM tests already work with automatic CI support, their code location is not satisfying according to maxim 1. I will propose an integrations of VM tests as a module option, producing outputs neatly piecing into the current hydra setup.