CppCon 2017: John Farrier “Performance Benchmarking with Celero”

—
—
Let's talk about performance optimization!

Developing consistent and meaningful benchmark results for code is a complex task. Measurement tools exist (Intel® VTune™ Amplifier, SmartBear AQTime, Valgrind, etc.) external to applications, but they are sometimes expensive for small teams or cumbersome to utilize. Celero is a small library which can be added to a C++ project and perform benchmarks on code in a way which is easy to reproduce, share, and compare among individual runs, developers, or projects.

This talk will start with an overview of baseline benchmarking, how proper measurements are made, and offer guidelines for performance optimization. It will then walk developers through the process of developing benchmark code in a way similar to many unit testing libraries. Through practical examples, methods for benchmark design and debugging will be explored. We will then use the library to plot and understand the results.

In the end, attendees should feel comfortable exploring the use of Celero in their own projects and adding baseline benchmarking to their testing and delivery processes.
—
John Farrier: Booz Allen Hamilton, Chief Engineer

John Farrier is a software engineer, researcher, and musician. He designs software architectures for military modeling and simulation activities. His projects support efforts across the U.S. Department of Defense ranging from lab-based experimental software to fielded software on live fire test ranges.
—

*-----*
*-----*

[frenchmarty]

35:34

Questionable reasoning here:

1.) The error, overhead and system terms should be the same (on average) for all of your tests. They may cause you to underestimate the relative improvement between implementations but they should not change **which** implementation is better (assuming you ran enough tests).

2.) What if I care about worst-case or 95th percentile performance?

3.) For a given sample size, uncertainty is much higher for the tails of a distribution than the mean. You are much more likely to get an anomalous result comparing the minimums than comparing the means of two sets. Algorithms that vary wildly in performance are also going to have lower minimums than algorithms with consistent performance.

If you're worried about outliers, remove the top and bottom X% of samples and calculate the trimmed mean. Accept that the relative differences are going to be understated (but not reordered) because of overhead.

[aliancemd]

42:53 - dude, that's not a good way to "sell" your library to people :D