CppCon 2018: Boris Kolpackov “What to Expect from a Next-Generation C++ Build System”

—
—
Few will argue that a standard build system would greatly benefit the C++ community. And the change is in the air: the upcoming C++ Modules are bound to shake things up, the need to support multiple build systems is putting pressure on our packaging efforts, and languages like Rust and Go clearly show things work much better if the build system and package manager are well integrated. Perhaps then this is the C++ community's "git moment": an opportunity to get a next-generation build system that everyone is finally happy to use.

In this talk I will demonstrate key features the next-generation C++ build system should have in order to achieve that elusive happy-to-use property. It is based on our experience developing the build2 toolchain, an integrated build system and package/project dependency managers (and which already supports C++ Modules).

We start with a brief discussion of overall design choices: native or project generator/meta build system, black box or a concept of build, declarative or scripted (or perhaps a hybrid), uniform or platform-specific, in which language can/should it be implemented, and, finally, what about IDE/tools integration?

Next, we go over what any current-generation build system should already support: build configuration management, cross-compilation, additional operations (test, install/uninstall, dist), out of source builds, and wildcard patterns.

Then on to the next-generation: high-fidelity hermetic builds, precise change detection, portable, parallel testing, project importing/composability, and support for C++ Modules which should pave the way to reliable, generally-available distribution compilation.
—
Boris Kolpackov, Code Synthesis
Chief Hacking Officer

Boris Kolpackov is a founder and CHO (Chief Hacking Officer) at Code Synthesis, a company focusing on the development of open-source tools and libraries for C++. For the past 10 years Boris has been working on solving interesting problems in the context of C++ using domain-specific languages (DSL), C++ parsing, source-to-source translation, and code generation. His notable projects to date include ODB (ORM for C++), XSD and XSD/e (XML Schema to C++ compilers), as well as libstudxml (XML parser/serializer library for C++).
—
*-----*

[DimiterStanev]

+1000 for requiring to explicitly list .h-eader files, and the fast pre-proc, post-proc scheme. Great talk. Thank you!

[DimiterStanev]

Boris is pointing out the "implementation language: Values" (20:59), but in my opinion the language, runtime, etc. and it's own values (java, python) should not conflate with the values of the final product. For example Google's bazel is written in java, and please.build is written in Go - but these systems (along with pants, gn, buck, etc.) do not target just one language (C/C++), but multiple - including python, bash scripts, etc. While bazel might be real slow the first time, since it keeps a server in memory, it can be very fast on subsequent runs. Also it doesn't have to use the standard java packaging, but it's own (e.g. it's own tool, that just happens to be written in java). I feel that this "raciality" where everything you do use must come, or be written in the main language you use (or the main programming, not scripting) is simply not beneficial. Yes, usually the contra argument is that - writing the build system in the language I know would help me change it easier, and that may be true - but the question you should be asking yourself is - how often would that happen to a typical user of that build tool.

[mapron]

54:30 - How to check your build system is working properly?I have one answer on that - you can easily check make -n output (or compiler database, if you have any). If build system for one particular configuration outputs just the same compiler and linker invocation (create a tool that compare arguments without checking order) - it probably correct. I do not talk about messing with invocation order - running yacc before compiling generated code, etc.

I have written cmake scripts to build ffmpeg with all dependencies, so I tested mostly two things: "make -n" are identical and all generated files are the same.
After that, I could not have a ready answer, how to check all possible configurations for new build systems, but I think it possible to create "configure tests" which check dry run output and generated files.

[TaiDesHen]

great talk, lots of good points few questions/comments from my point of view if someone knowledgable about build2 will read this: :)

1. We are migrating to conan.io in my company. Are there any plans and even is that a possibility to use both conan.io and build2? Conan.io to manage deps and build2 as a build system.

2. Project composability, subprojects etc, Not sure if this is what I am thinking of, but in conan.io workspaces are now being introduced, which is a feature i fell in love with while working in rust. Can you please link or tell me more about this topic?
My most common scenario in c++ right now is as follows:
I am working on a project, this project contains multitude of lirbaries. every library is located on separate git repository. Before workspaces each time i made a change to library i had to deploy project in order to use that change in upstream projects, now with conan.io workspaces i am able to work on workspace made of 10+ projects, which is super great. I achieve that doing git submodules and workspace configuration from conan.io. Upon changing library binary detects that change and recompiles everything as needed. is this something that you were talking about? in b2 is it possible to use submodules like that?

3. Wildcards - always tough topic. :) I am using cmake globs, even though they are not recommended, because its cmake everytime i add file /change branch /rename/remove i have to reload cmake. From what I understood b2 supports wildcards so in my opinion thats great. Meson build system is against wildcards, but honestly I think for most of the project the cons of having these are outweight by pros - not having to look at the build file multiple times a day is great, not to mention conflicts on file lists...

4. Unfortunatelly i will not be pushing b2 as a backend even if it worked with conan.io because of lack of IDE support.
I am using clion now, and its getting better, finally using clang as a backend for understanding code.

[Hoowwwww]

for build systemt to succeed it needs to come with the language out of the box
i wanna try c++ ? install cppbuild, and done, you can create project, add package, test, etc
new script language ? NO, i don't want to learn new language, either you use c++ or use data json/xml

[AnonEMuss-gwfm]

IDE support in general, and Visual Studio support in particular, is critical to the success of any new build system.

There are a huge segment of C++ programmers who live in Windows-land, and you need to have *excellent* VS support to even be considered. Many of these programmers will *also* want a great CLI, but it has to be optional.

Also... IDE integration doesn't just mean pressing the "build" button and having it run your build tool. Project management needs to be fully integrated into the IDE GUI. Editing build files and manifests in the IDE text editor doesn't count as "integration"! One consequence of this is that build files need to be both human and programatically editable. Makefiles and CMakeLists.txt are awful in this regard -- there's no way for an IDE to safely update them. JSON (or XML) work much better.