In this post I want to describe how you can setup an automated build for Visual Studio C++ projects using the build toolchain based around maven.
Disclaimer: I'm aware of Microsofts's Team Foundation Server ALM stack. It is a powerful and wonderful toolsuite for software development. On the other hand, if you don't want (or can't afford) TFS, the presented approach may be an alternative.
| java build in c++ land |
Disclaimer: I'm aware of Microsofts's Team Foundation Server ALM stack. It is a powerful and wonderful toolsuite for software development. On the other hand, if you don't want (or can't afford) TFS, the presented approach may be an alternative.
Goals
- It should be possible to compile Visual Studio projects on a build server
- The build should be automated
- Tests should be executed alongside with compiling the code
- Compiled artifacts should be delivered to the developers on demand
- There should be a minimal impact for the workflow of the C++ developers
- Code should be partitioned into modules
- Modules have a public interface
- Different modules can have different release cycles
- If there are build errors or test failures, the development team should be informed
- ...
This sounds like a continuous integration story - in the Java world it would be a no brainer - just use Jenkins, Maven, a server to distribute your artifacts and a Unit Testing Framework - everything is set up and ready to go.
As it turns out, this can also be true for compiling C++ using Visual Studio projects. The only difference is that you have to know how to configure the maven build - that is to say how to organize your source code and tinker with some xml files. It imposes some work on the initial setup, but after that it works quite well, if not good enough.
A cornerstone of this approach is that the workflow for the C++ guys is the same as if the whole build process would not exist. What I mean here is that the C++ experts can configure their solution in (almost) any way they want, and use the tool they are comfortable with for doing so: Visual Studio. Likewise, they shouldn't be bothered configuring anything related to the build process.
The only concession they have to make is to understand the basic maven workflow (mvn install, mvn generate-resources) which can be hidden behind some custom tool actions in VS.
Because of those considerations, I won't involve the otherwise well suited Maven NAR plugin for the build, since this would involve configuring dependencies on project level - this would interfere with the configuration in the solution files. Moreover, this plugin supports OS independent builds, but this is not a goal which I'm pursuing here.
Suppose you have two applications which consist of three modules. App 1 only needs module A and has some extra functionality. App 2 on the other hand has three modules (module A, B, and C) and also its own program logic. Let's assume that app 2 is more complex than app 1, but still they share the same code (module A).
Code sharing using C++ can be either done by linking it statically or dynamically. In the first case, you will have lib files, which end up in the executable, or, in the second case, you have dynamic link libraries (dlls) which can be shared for more than one application (you just have to make sure that they can be found by the exe file at runtime.)
Typically, all components have a development lifecycle of their own - meaning for example that module A has to be extended for app 2, since requirements may have changed. App 1 doesn't need those changes, it would happily work with the "old" version of module A. Even worse, app 1 may not even work with the changes made to module A.
This describes the general problem that certain parts of your software stack have different release cycles. One module may not change for weeks or months (then it is either a very well written module or just plain uninteresting ;-) ), other modules are hotspots of activity. A mechanism to reference a module in a certain version would be very handy.
You can address the versioning issue by branching and tagging module A source code and check out the appropriate version and compile module A every time (or at least once thanks to the incremental compile features of Visual Studio).
However, there may be other constraints which make this approach not feasible, like needing a special compile step which has high demands on the machine or licensing costs for involved compilers. Maybe the compilation of the dependencies just takes too long - you get the idea.
All in all, you are just interested in linking module A and be able to interface it - the compilation step for module A should be done by the domain expert who knows all the quirks to compile or maintain it. Ideally and in general the build should be automated and run on a mouse click. The binaries should be downloaded from a repository - maven and it's infrastructure are very good with those things.
The idea is now that you create a pom which delegates the compile and test phase to the cpp compiler, and fetches its dependencies with the maven dependency plugin. Typically, the dependencies have to be fetched in both Debug and Release mode, alongside with pdb's or other compile artefacts you'll need to be able to create or debug the final application.
After some experimentation I came to the conclusion that you get the best results if you define that every solution is a module on it's own (each vcxproj should be part of only one solution). For every solution file you define one pom which is used to describe the module's dependencies and it's artifacts which the compile step is producing at the end of the day for the given solution.
As mentioned above, you'll need Visual Studio, Java, Maven and a versioning system (git, subversion) on the developer machines. On the build server you should install Jenkins and also Visual Studio. Finally Nexus would be a good choice for distributing the artifacts.
External dependencies (libs, dll's and header files) can be put into a zip file with a decent directory structure and uploaded to the nexus.
The figure above depicts a possible structure for the example application. You can see that for every module you have to define a pom file, and also every application has a pom file. Also every module or application has one solution file. There is the convention that the solution file of a module only references project files which are contained within this module. This makes it possible also to take advantage of the release process which comes for free with maven.
Every module has an interface (header files) and a file which defines what are the output artifacts of this specific module. The details are all to be configured using the facilities which are available in the solution files. Like this every developer who is owner of a module can setup it's specific compilation without having to know specifics about maven.
An advantage of this is that you can work with different versions of the modules, and the dependencies are both documented and part of the build by residing in the pom.xml files.
Like mentioned above a module should not reference any files outside of its scope. This means that for example source code in module-b-p2 can reference source code in module-b-p1 using relative paths (although I would say that this is bad practice) - but it is not allowed for module-b source code to reference module-a source code directly - this would break the module barriers.
The question is: how can we now reference from one module to the other? The trick is now that module artifacts (libs,dlls,pdbs and interface) are fetched by the maven dependency plugin to the target folder of a module, and by exploiting the $(SolutionDir) variable which is available in Visual Studio we have something similar like the ${project.basedir} in maven.
Lets have a look at following figure:
What one can see above is that there is a new folder in the app1 directory called 'target' where, by convention, all build artifacts, temporary files and dependencies for the app1 build are stored. By configuring the maven dependency plugin in a certain way it is quite trivial to put the build output from module-a in the target\deps\$(Configuration) directory. If you configure additional link library directories and special include directories in Visual Studio, the compiler will happily compile your files.
By convention, the build outputs should be placed under target\rt\$(Configuration) folder. To be able to properly debug the application, the build process should also place all runtime dependencies to this directory.
Building applications in a heterogeneous environment is not always easy, everybody has to leave his comfort zone and adopt something new. For programmers who are not acquainted with the maven build system (and I assume most of the C++ crowd doesn't know about it) it is better not to make them type in things like "mvn clean" or "mvn package", they want a better integration in their IDE.
Luckily enough you can customize Visual Studio in many ways, and one way I would suggest to integrate maven commands as easily accessible buttons in their IDE. This can be done by configuring this once per seat.
This custom tool can be placed on a button which essentially reduces all maven magic to one mouse click for the uninterested developer. If you have three of them (one for mvn install, one to "fetch dependencies" and additionally one for clean up (mvn clean)) the devs will be happy. The normal workflow for a developer will then be an update of the code with the given versioning system and a mouse click to get the newly built dependencies.
I don't really think it is. If you follow certain conventions like outlined in this post, creating new modules becomes a fairly easy task. Typically, you won't introduce new modules every day.
Depending on the size of your development team, only the seniordictator developer will decide when to jump to a new version of a module with potential breaking changes. In fact, pinning down dependencies like that is very successful in java land, so why shouldn't it make sense also for c++ projects?
One should not forget that, leaving those module versionings and module depency definitions aside, nothing really changes for the average developer. Typically, most of the devs work in one module (which may contain dozens of subprojects which may be arbitrarily complex), and only the tech leads compose modules together.
If you do it right you get goodies like being able to release your cpp code with the maven release plugin. This is a huge win and definitely worth the trouble of setting up the initial build. You could profit by the plethora of possibilities which the maven ecosystem gives - for example easy filtering of resources, arbitrary plugins, reporting (for example integrate doxygen reports in your build or use the not so popular but still very cool "site" feature for writing versionable documentation) ... and it's free.
I hope someone finds this useful, thanks for reading!
As it turns out, this can also be true for compiling C++ using Visual Studio projects. The only difference is that you have to know how to configure the maven build - that is to say how to organize your source code and tinker with some xml files. It imposes some work on the initial setup, but after that it works quite well, if not good enough.
A cornerstone of this approach is that the workflow for the C++ guys is the same as if the whole build process would not exist. What I mean here is that the C++ experts can configure their solution in (almost) any way they want, and use the tool they are comfortable with for doing so: Visual Studio. Likewise, they shouldn't be bothered configuring anything related to the build process.
The only concession they have to make is to understand the basic maven workflow (mvn install, mvn generate-resources) which can be hidden behind some custom tool actions in VS.
Because of those considerations, I won't involve the otherwise well suited Maven NAR plugin for the build, since this would involve configuring dependencies on project level - this would interfere with the configuration in the solution files. Moreover, this plugin supports OS independent builds, but this is not a goal which I'm pursuing here.
Example scenario
 |
| programmers like rectangles and arrows |
Suppose you have two applications which consist of three modules. App 1 only needs module A and has some extra functionality. App 2 on the other hand has three modules (module A, B, and C) and also its own program logic. Let's assume that app 2 is more complex than app 1, but still they share the same code (module A).
Code sharing using C++ can be either done by linking it statically or dynamically. In the first case, you will have lib files, which end up in the executable, or, in the second case, you have dynamic link libraries (dlls) which can be shared for more than one application (you just have to make sure that they can be found by the exe file at runtime.)
Typically, all components have a development lifecycle of their own - meaning for example that module A has to be extended for app 2, since requirements may have changed. App 1 doesn't need those changes, it would happily work with the "old" version of module A. Even worse, app 1 may not even work with the changes made to module A.
This describes the general problem that certain parts of your software stack have different release cycles. One module may not change for weeks or months (then it is either a very well written module or just plain uninteresting ;-) ), other modules are hotspots of activity. A mechanism to reference a module in a certain version would be very handy.
You can address the versioning issue by branching and tagging module A source code and check out the appropriate version and compile module A every time (or at least once thanks to the incremental compile features of Visual Studio).
However, there may be other constraints which make this approach not feasible, like needing a special compile step which has high demands on the machine or licensing costs for involved compilers. Maybe the compilation of the dependencies just takes too long - you get the idea.
All in all, you are just interested in linking module A and be able to interface it - the compilation step for module A should be done by the domain expert who knows all the quirks to compile or maintain it. Ideally and in general the build should be automated and run on a mouse click. The binaries should be downloaded from a repository - maven and it's infrastructure are very good with those things.
The idea is now that you create a pom which delegates the compile and test phase to the cpp compiler, and fetches its dependencies with the maven dependency plugin. Typically, the dependencies have to be fetched in both Debug and Release mode, alongside with pdb's or other compile artefacts you'll need to be able to create or debug the final application.
After some experimentation I came to the conclusion that you get the best results if you define that every solution is a module on it's own (each vcxproj should be part of only one solution). For every solution file you define one pom which is used to describe the module's dependencies and it's artifacts which the compile step is producing at the end of the day for the given solution.
Prerequisites
As mentioned above, you'll need Visual Studio, Java, Maven and a versioning system (git, subversion) on the developer machines. On the build server you should install Jenkins and also Visual Studio. Finally Nexus would be a good choice for distributing the artifacts.
 |
| upload to nexus is easy |
External dependencies (libs, dll's and header files) can be put into a zip file with a decent directory structure and uploaded to the nexus.
How to setup your source code
 |
| example setup for the code organisation |
The figure above depicts a possible structure for the example application. You can see that for every module you have to define a pom file, and also every application has a pom file. Also every module or application has one solution file. There is the convention that the solution file of a module only references project files which are contained within this module. This makes it possible also to take advantage of the release process which comes for free with maven.
Every module has an interface (header files) and a file which defines what are the output artifacts of this specific module. The details are all to be configured using the facilities which are available in the solution files. Like this every developer who is owner of a module can setup it's specific compilation without having to know specifics about maven.
An advantage of this is that you can work with different versions of the modules, and the dependencies are both documented and part of the build by residing in the pom.xml files.
How are module dependencies defined in the solution files?
Like mentioned above a module should not reference any files outside of its scope. This means that for example source code in module-b-p2 can reference source code in module-b-p1 using relative paths (although I would say that this is bad practice) - but it is not allowed for module-b source code to reference module-a source code directly - this would break the module barriers.
The question is: how can we now reference from one module to the other? The trick is now that module artifacts (libs,dlls,pdbs and interface) are fetched by the maven dependency plugin to the target folder of a module, and by exploiting the $(SolutionDir) variable which is available in Visual Studio we have something similar like the ${project.basedir} in maven.
Lets have a look at following figure:
 |
| app1 folder after mvn initialize |
By convention, the build outputs should be placed under target\rt\$(Configuration) folder. To be able to properly debug the application, the build process should also place all runtime dependencies to this directory.
Example pom file
Example artefact assembly file
Example interface assembly file
Screenshots of Visual Studio projects (properties)
 |
| General configuration properties for a Visual Studio project |
 |
| linker include paths have to be adapted |
How to get acceptance in the development team
Building applications in a heterogeneous environment is not always easy, everybody has to leave his comfort zone and adopt something new. For programmers who are not acquainted with the maven build system (and I assume most of the C++ crowd doesn't know about it) it is better not to make them type in things like "mvn clean" or "mvn package", they want a better integration in their IDE.
Luckily enough you can customize Visual Studio in many ways, and one way I would suggest to integrate maven commands as easily accessible buttons in their IDE. This can be done by configuring this once per seat.
 |
| Tools -> External Tools : configure maven |
This custom tool can be placed on a button which essentially reduces all maven magic to one mouse click for the uninterested developer. If you have three of them (one for mvn install, one to "fetch dependencies" and additionally one for clean up (mvn clean)) the devs will be happy. The normal workflow for a developer will then be an update of the code with the given versioning system and a mouse click to get the newly built dependencies.
But it is so much overhead!
I don't really think it is. If you follow certain conventions like outlined in this post, creating new modules becomes a fairly easy task. Typically, you won't introduce new modules every day.
Depending on the size of your development team, only the senior
One should not forget that, leaving those module versionings and module depency definitions aside, nothing really changes for the average developer. Typically, most of the devs work in one module (which may contain dozens of subprojects which may be arbitrarily complex), and only the tech leads compose modules together.
Advantages
If you do it right you get goodies like being able to release your cpp code with the maven release plugin. This is a huge win and definitely worth the trouble of setting up the initial build. You could profit by the plethora of possibilities which the maven ecosystem gives - for example easy filtering of resources, arbitrary plugins, reporting (for example integrate doxygen reports in your build or use the not so popular but still very cool "site" feature for writing versionable documentation) ... and it's free.
I hope someone finds this useful, thanks for reading!
Thanks for taking time and posting a step by step guide lines. I never used Maven but seems like after this I might give it a try at my company.
ReplyDeletethank you! :) good luck!
DeleteHow do you separate the compile and test phases? It would see to me you have to move the solution and vcxproj files down to src/main and src/tests so that the two phases are completely separate. I guess another option is to have a test task. I'm looking into this "issue" at the moment.
ReplyDeleteAnyway, your post was very helpful, although I couldn't convince the team to go to my preferred solution: MingW :-)