Well, I got pushed, so now I have just delivered to my team probably the first SCons replacement for MSBuild, meaning that I use MSBuild configuration files to build rather than normal SCons SConscript.py files, although I can do both simulataneously. The developers get to use both build systems, and use Visual Studio to actually build and manage the build. What I have over on MSBuild is that I do not have the Solution boundary that MSBuild does, so I can build many Solutions simultaneously and at the individual target level, not at the project level like MSBuild. Our build right now consists of 53 Solution files and 3 SConscript.py files, and hundreds of VCXproj and CSproj files. Oh, and we build Debug and Release simultaneously as well.

As for performance, I have not measured as closely as you, so no pretty graphs, but I think I am now seeing how SCons is getting in its own way, but we are going so fast no one is looking, well, except for me. Our typical build machine is now a 3.33 GHz machine 6 cores with 12G of ram. Why 12G, well you said it, on a Win7 box with the size of our build, SCons is using up about 6G, which is what we started with, and a few noticed great build times, but the machines were not very responsive during a build. The 12G made the machine responsive again, and sadly, no improvement in build times.

Hyperthreading on or off doesn’t manner, the processor seems to be in our way as we are doing full builds incredibly fast, typically 12 tasks at a time, with hyperthreading on, across all boundaries that MSBuild is not capable of doing. We are building approximately 20,000+ targets in about 70 minutes, but here is where I believe you have hit the nail on the head. I believe the actual build time could be much faster as SCons is starving the cores by not being able to find enough things to build fast enough. That starvation is from SCons not being able to multithread its own internal work to find things to build faster. If it could farm out the scanning of dependencies to other processes/threads and in that way it could build the dependencies tree faster and provide targets to be built faster. Another measure of how well the build is doing, but yet SCons internally is not doing well is that the gap between a do nothing build and a full clean build has closed considerably, where before on single threaded builds the do nothing builds would be a small fraction of the total build time, and now, the incremental do nothing build is greater the 50% of the total build time.

SCons is scanning for dependencies on the fly, therefore removing a ton of developer maintenance and errors, and yes, that costs, but it costs more and is less accurate for the developer to do it. We are getting that done reliably and I have a lot of happy developers. Everything right now is how do we multi-process more tasks and save yet more time.

So, I hate to be argumentitive, but scale it does, very well, I have not detected any limit that at least cheap memory will not solve nicely and they are already shipping 8 core processors. And as builds get large, they would normally get harder to maintain, but this is all the more reason to use SCons where dependency scanning shines in a big way.

So, given a total choice of build systems would I use SCons again? Well, so far my boss keeps denying me the freedom to write my own, which I would prefer to write that side now that I have the MSBuild side done. And even if I did, and wrote it much better with more multiprocessing, laws of deminishing returns, I believe we are running tight enough that we would likely not get the gain we would hope for, and again, we are blazing right now. Faster processors, now there is where it is at.

Regarding the use of ‘toy’ sources — obviously that’s what I had to do for these tests. There aren’t many real projects of the magnitude I was testing (tens of thousands of files); at least, there aren’t many that are publicly accessible. Even if such a project existed, it’s not worth my time to convert the build system of a large real project from one tool to another simply for the sake of benchmarks like this.

Finally, when you’re talking about a substantial code base, 10% is still a significant amount of actual time. On a two-hour long build, that’s almost 15 minutes. Not huge, but not chump change either. Given my druthers, I’ll take the faster build system and get out of the office 15 minutes before you every day .

srcs = [big list o' files]
targets = [big list o' files]

and then naively did

env.SomeBuilder(targets, sources)

SCons makes every target in a builder invocation depend upon every source, so rather than the dependency graph gaining N edges, it gains N^2 edges (assuming, for example, the sources are C files and the targets are their corresponding .o files).

The shortcuts of either inserting a proxy dependency (one of the tricks from the “GoFastButton” page) or using iteration or a list comprehension to invoke the builder once per source file/target file pair both result in DRAMATIC improvements in performance.

I have not yet looked at your benchmark, but I can say, having converted a code base of about 2M lines of code, that scons is actually faster than GNU make in our build, when building (or incrementally rebuilding) the whole tree, and in particular when using a cache for rebuilds (10x speedup there).

Furthermore, one cannot understate the improvement that comes from using an MD5 hash to decide when to rebuild: the ripple effects of making a change are MUCH better contained, as you no longer necessarily have to rebuild EVERYTHING that’s downstream of your change.

I hope to have time to look at your benchmark in detail; thus far, all of the benchmarks that I’ve seen have used toy sources that are pathological for scons: sure, the overhead can be huge when the compile time for each source file is on the order of a fraction of a second. On a real code base, however, the compile times aren’t necessarily small (template-heavy C++ code built with -O2, for example, can take awhile), and the overall speedup you get from limiting downstream ripple effects from a change, coupled with the ability to pull stuff from the cache, ends up being a HUGE win. IOW, the overhead from SCons on nontrivial code bases, although still observable, fades to something less than ten percent, IME.

Thanks for the article.

http://www.scons.org/doc/2.0.0.final.0/HTML/scons-man.html
Yeah it wasn’t clear