Can SCons “GoFast”?

You can read all about the setup in the previous post, so I’ll just jump straight to the results. According to the “GoFastButton” recommendations, –max-drift=1 –implicit-deps-unchanged will “run your build as fast as possible”, so that’s what I used. In all cases, I did an initial, untimed from-scratch build first, to generate the initial dependency graph, then I ran a second timed build. After that timed run, I ran a clean build and finally ran the full build again, this time with -j 2, to evaluate the impact of SCons parallel build features.

Contrary to my expectations, the GoFast settings had relatively little impact over much of the test range — only about 5-10% faster than without those flags. Only the very largest build showed any significant impact, with a 25% improvement. Unfortunately that impressive result is more likely because SCons uses less memory with GoFast settings enabled. If you recall from the previous tests, with 50,000 source files, SCons’ memory footprint was a hefty 2,023 MB — enough to force my test machine to start swapping. With the GoFast settings, SCons used “only” 1,838 MB — still an awful lot of memory, but just smaller enough to avoid thrashing the system, with the end result being a substantially improved build time.

Building in parallel had a more substantial impact — reducing build times about 30% on the largest build (compared to a serial SCons build with GoFast settings enabled). That’s not as good as I had hoped for (on a large, relatively “flat” build such as this, I expected the build to parallelize very well), but it’s not terrible. Here are the complete results:

So, GoFast seems to be a bust on full builds. It’s definitely better than vanilla SCons, but still nowhere near as fast as gmake.

Things look a little better on “one-touch” incremental builds though, where GoFast settings cut build times by about 30% across the board:

Ironically, the most impressive results are on clean builds (scons -c). GoFast settings cut build times by about 40% at the low end of the test range, and by more than 50% at the high end of the test range:

To my amazement, SCons with GoFast settings actually beats gmake on clean builds. My guess is that this is probably because SCons handles file deletion in-process, while gmake must invoke a separate process (rm).

That’s all folks!

That’s it for my analysis of SCons performance. Thanks to everybody who offered ideas for improving my benchmarks! I hope you found this series of posts interesting.

.ONESHELL

Normally, gmake executes each line of a rule body or recipe using a separate invocation of the shell. With gmake 3.82, you can add the special target .ONESHELL to the makefile, which will tell gmake to run the entire recipe using a single shell invocation. For example, if your makefile contains the following:

all:
	@export FOO=1
	@echo FOO is -$$FOO-

Without .ONESHELL, gmake will invoke the shell twice, as follows:

sh -c 'export FOO=1'
sh -c 'echo FOO is -$FOO-'

Naturally, that will not produce the output you actually want (ie, “FOO is -1-”). With .ONESHELL, gmake instead invokes the shell just once:

sh -c 'export FOO=1
    echo FOO is -$FOO-'

In addition to making it easier to write multi-line rule bodies, this feature makes it much more palatable to use alternative shells. For example, you could imagine using Perl as the shell:

SHELL=perl
.SHELLFLAGS=-e
.ONESHELL:

all:
	@my $$foo = "1";
	print "FOO is -$$foo-\n";

(Note the use of another 3.82 feature, .SHELLFLAGS, which allows us to control the command-line flags used with the shell; in this case I’ve set those to “-e”, the Perl flag for executing a script from the command-line).

Pattern search changes

Prior to version 3.82, when gmake finds multiple matches during a pattern search, it prefers patterns declared earlier in the makefile over patterns declared later. As of 3.82, gmake instead prefers the pattern that results in the shortest stem. That sounds a bit confusing thanks to the jargon, but I think this will actually cause gmake to better adhere to the principle of least astonishment. Here’s an example:

all: sub/foo.x

%.x:
	@echo "Prefer first match (stem is $*)."

sub/%.x:
	@echo "Prefer most specific match (stem is $*)."

Compare the output from gmake 3.81 and 3.82:

Prefer first match (stem is sub/foo).

Prefer most specific match (stem is foo).

gmake 3.82 prefers the second pattern because it is a more specific match than the first. Note that this is a significant backwards-incompatibility compared with previous versions of gmake!

Other changes

Besides those big changes, there are several smaller features, such as:

.RECIPEPREFIX

This special variable allows you to change the character used to mark the beginning of a command in a recipe from the default TAB character. I can only assume that the developers added this feature to quell the semi-regular complaints about make’s sensitivity to whitespace.

private variable modifier

Normally, gmake propagates target-specific variable assignments to the prereqs of the target. With the private modifier, you can restrict the scope of a target-specific variable assignment, so that it is not inherited by the prereqs.

undefine directive

The inverse of the familiar define directive, undefine lets you completely remove a variable definition.

define improvements

The define directive now supports the same assignment operators that regular variable assignment alows: :=, ?= and +=, for simple, conditional and appending assignments.

If you want to see the full list, you can find it in the NEWS file in the gmake source tree.

“Rumors of my death have been greatly exaggerated…”

Some naysayers claim make is outdated, but it’s clear that make is still alive and kicking (and if a new gmake release isn’t proof enough, take a look at some of the innovations we’ve put into Electric Make). A hearty congratulations to everybody who contributed, and especially to Paul Smith for driving the development and release effort. Keep up the good work!

A few months ago, I took a look at the scalability of SCons, a popular Python-based build tool. The results were disappointing, to say the least. That post stirred up a lot of comments, both here and in other forums. Several people pointed out that a comparison with other build tools would be helpful. Some suggested that SCons’ forte is really incremental builds, rather than the full builds I used for my test. I think those are valid points, so I decided to revisit this topic. This time around, I’ve got head-to-head comparisons between SCons and GNU make, the venerable old workhorse of build tools, as I use each tool to perform full, incremental, and clean builds. Read on for the gory details — and lots of graphs. Spoiler alert: SCons still looks pretty bad.

The setup

As before, my test system is a dual 2.4 GHz Intel Xeon, with 2 GB RAM. But since the previous set of tests, my system has been upgraded to RHEL4; there’s a new version of Python; and there’s even a new version of SCons — the big 2.0. So the setup is now:

• RedHat Enterprise Linux 4 (update 8, kernel version 2.6.9-89.ELsmp)
• Dual 2.4 GHz Intel Xeon, with hyperthreading enabled
• 2 GB RAM
• Python 2.7
• SCons v2.0.0.final.0.r5023
• GNU make 3.81

As previously, the test builds consists of a bunch of compiles and links: N C files, each with a unique associated header, spread across N/500 directories (to ensure there are no filesystem scalability effects), are compiled, then bundled into a standard archive library. Every 20th object is linked into an executable along with the archive. The build tree is generated using a Perl script, which generates both SConstruct files and Makefiles for building. One difference between the two builds: when using GMake, I added the -MMD flag to gcc, to generate additional dependency information; each timed full build was then preceded by a full build to generate all the dependencies and a clean build to nuke all the generated output. I felt that this gave a more realistic comparison to SCons, which employs elaborate dependency analysis logic to ensure accurate incremental builds.

Round 1: Full builds

Again, my interest is primarily full, from scratch builds:

As expected, SCons’ performance is pretty miserable: by the time we get to a build with several thousand source files, the build time is already over 15 minutes, and with the same n² growth we saw last time, the times race past one hour, then two, finally hitting nearly 5 hours for a build containing just 50,000 source files.

GNU make shows a much more sedate, linear growth — in fact it neatly keeps pace with a simple shell script that does all the same compiles and links as the regular build, but without any of the overhead of a build tool. Yes, I know that a shell script is no substitute for a proper build tool. It just serves to give us an idea of how long it takes just to do the work in the build — a lower bound on the build time. No build tool running serially can run the build any faster than this (obviously we can do better, by a constant factor, if we use parallel build features).

Using that lower bound, we can compute the amount of overhead introduced by the build tool. This is all the time that the tool spends doing things other than actually running the commands needed to execute the build: parsing SConstruct files or makefiles, building the dependency graph, traversing that graph, computing command-lines, etc. Viewing this overhead as a percentage of the total build time gives us an easier-to-digest comparison between SCons and GMake:

Even with relatively few files — around 2,000 — over 50% of the total build time is wasted by overhead with SCons. In comparison, GMake has barely any overhead on a build of that size. At the other end of the range, SCons overhead accounts for nearly 90% of the total build time. GMake overhead has increased as a percentage of the total time too, but only to a modest 20% — still significantly less than SCons. In fact, with 50,000 files, GMake overhead is less than half of SCons overhead with just 2,000 files!

OK, just a couple more things to touch on here before we look at incremental build times: first, you probably noticed the sudden hook in the graph for SCons full build times, between 45,000 and 50,000 files. That means that at that level, on my system, some other factor has kicked in to influence the times. I believe this is because the system has started to page heavily, as SCons’ memory footprint has grown to consume all available RAM. Second, if you compare the SCons times in this article with those in the previous article, you’ll see that this time around, the SCons times are a bit better than last time — about 30% on average. Keep in mind that there are several differences between this setup and the previous: a new OS, a new version of Python, a new version of SCons. Even a new version of the compiler and other tools used during the build. I did try to pin down exactly which factors contributed to this improvement by testing various combinations of versions of Python and SCons (for example, Python 2.6.2 with SCons 2.0; or Python 2.7 with SCons 1.2.0); none of these tests produced any significant change in performance, so I presume that the performance difference is most likely due to the new operating system and tools. I choose not to pursue the matter further than that.

Round 2: Incremental builds

A lot of people claimed that full builds “don’t really matter; developers only really do incremental builds”. So let’s take a look at incremental build performance:

For these builds, I ran a full build, then patched one C file and its associated header. This caused a rebuild of one object file, followed by a rebuild of the archive and the executable that the object feeds into. The actual time to execute just those commands is a paltry one-tenth of a second, so with either tool, the incremental time is dominated by the overhead added by the tool itself. Even so, it’s obvious that SCons adds considerably more overhead than GMake. Even with a small build containing just 2,000 files, SCons burns about 35 seconds to do one-tenth of a second of work. GMake does the same build in about 3 seconds. For the 50,000 file build, SCons ran for about 25 minutes, again just to do one-tenth of a second of actual work; GMake ran for about 9 minutes.

One thing I find especially interesting about this graph is that supposedly the problem with full builds is that they force SCons to constantly rescan the dependency graph. That shouldn’t be necessary in an incremental build, and yet we still see what looks like an O(n²) growth in build time.

Round 3: Clean builds

The last build comparison was a clean build: scons -c versus gmake clean:

At last we have found a build variant where SCons performance is in the same ballpark as GMake! Of course, according to the SCons docs, you’ll never need to do a clean build with SCons (because with SCons your dependencies are perfectly accurate), so maybe this data point is not actually interesting to SCons users.

Sudden death overtime: Memory usage

One last comparison: memory usage. This metric is of particular interest because it puts a hard limit on the maximum size of the build that the tool can handle. I’ve learned the importance of memory efficiency the hard way — long nights in “firefighting” mode to improve memory usage to accomodate this or that customer’s enormous build. I’d hoped that the new versions of Python and SCons used in this test would prove beneficial for SCons memory footprint. Unfortunately, the opposite is true: memory usage is about 8% worse now than it was the last time I ran these tests, and as you can see here, SCons uses about 4 1/2 times as much memory as GMake:

With 50,0000 files, SCons uses just a hair less than 2GB of memory, enough to cause my test system (with 2GB of RAM) to start swapping. In comparison, GMake needs just 440MB of memory for 50,000 files. At that rate, GMake won’t start to thrash my system until the build has grown to more than 225,000 files. Unfortunately, it seems likely that there’s not much that the SCons developers can do to fix this: because SCons is implemented in Python, they are at the mercy of the Python runtime implementation. There’s just not enough control over low-level details like memory allocation when you’re using an interpreted language.

Conclusions

We can clearly see now that it is not simply “the nature of the beast” that SCons performance is so bad. GMake scales much more gracefully, both in terms of elapsed time, and in memory usage, on a variety of scenarios: full, one-touch incremental, and clean builds. From the discussions that the previous post sparked, we know that the primary problem is an inefficient O(n²) algorithm in the SCons implementation. It seems that SCons is academically interesting, but ultimately not practical for any non-trivial build (as some of my customers are now finding out the hard way).

A lot of SCons fans, in defense of SCons, say things like “Well, the SCons developers made a concious decision to focus on correctness rather than performance.” But what good is a correct system that is so slow it’s unusable? For me, the choice is a no-brainer: my time is too precious to waste on a slow build tool.

Of course, high performance is the entire raison d’être for ElectricAccelerator. We knew from the start that parallelism would be the primary means of achieving our performance goals (although it’s not the only trick we used). Thanks to Amdahl’s law, we know that in order to accelerate a build by 100x, the serialized portion cannot be more than 1% of the baseline time. Thus it’s critical that absolutely everything that can be parallelized, is parallelized. And I mean everything, even the stuff that you don’t normally think about, because anything that doesn’t get parallelized disproportionately saps our performance. Anything that isn’t parallelized is a bottleneck.

Here’s an example: command expansion. You know, the bit of code that turns something like this:

$(CC) $(addprefix -I,$(dir $(SRCS))) -o $@ $<

into something like this:

gcc -I. -Isubdir -o foo.o foo.c

There’s no way around this translation. It has to be performed, for every command invoked during the build. Even if the command doesn’t have anything that needs to be expanded.

What happens if the expansion itself is time-consuming? For the sake of demonstration, we can make command expansion slow by sticking $(shell sleep 5) into the command — because the sleep appears inside a $(shell) function call, gmake is obliged to execute the sleep as part of expanding the command. Here’s the makefile we’ll use:

all: a b c d e
a b c d e:
        @$(shell sleep 5) echo $@

Now, if you run this with an ordinary, serialized gmake, you’ll see that it takes about 25 seconds to execute. No surprise there, with 5 jobs that each have a 5 second sleep:

ericm@chester$ time gmake
a
b
c
d
e

real    0m25.092s
user    0m0.012s
sys     0m0.012s
ericm@chester$

Now, see what happens when we run this same makefile with a parallel gmake invocation: it still takes 25 seconds, even though we have specified more than enough parallel jobs that all five jobs should run simultaneously!

ericm@chester$ time gmake -j 8
a
b
c
d
e

real    0m25.016s
user    0m0.012s
sys     0m0.012s
ericm@chester$

What’s going on here? Let’s take a quick look at the core algorithm in gmake:

1. Find the next target that’s is runnable (all prereqs up-to-date).
2. Expand the commands for that target.
3. Run the command for that target.
4. If the number of currently running commands is equal to the job limit (1 for serial gmake, N for parallel gmake), wait for a command to finish.
5. Repeat until finished.

Maybe you see the problem already: gmake is a single-threaded program. Even when you specify -j 8, there’s only one thread executing that core algorithm. Parallelism doesn’t really enter the picture until the end of the algorithm, where gmake decides whether it can go ahead and start working on another target without waiting for the previous one to finish.

Being single-threaded means that command expansion is implicitly serialized: gmake can only expand commands for a single target at a time. Too bad for you if that expansion takes any significant amount of time.

So, could we fix gmake, so that command expansions could be performed in parallel? Well, it’s really hard to take something that wasn’t designed to be high-performance from the start and transform it into something with world-class performance. In this case, gmake’s heritage as a single-threaded application permeates every aspect of its implementation. For example, command expansion is performed using a single global buffer (see expand.c in the gmake sources). While it’s certainly possible to refactor this code, it would be non-trivial to do so.

Command expansion with ElectricAccelerator

In contrast, Accelerator was designed to be multi-threaded from the start, and we have made a deliberate effort to parallelize absolute everything we can, including even the behind-the-scenes stuff that you probably never thought about before reading this blog. Like command expansion. And sure enough, if you try this makefile with Accelerator, the total run time is about 5 seconds:

ericm@chester$ time emake
Starting build: 12705
a
b
c
d
e
Finished build: 12705 Duration: 0:05 (m:s) Cluster availability: 100%

real    0m5.523s
user    0m0.008s
sys     0m0.016s
ericm@chester$

Designing for high performance

When truly world-class performance is your goal, you can’t wait until implementation is complete to start thinking about performance. That goal has to inform your decisions at all stages of development, from design to implementation. With Accelerator, our obsessive focus on performance has resulted in an architecture that allows us to parallelize parts of the build process that other tools simply cannot.

One idea is to generate random text and use that as a “makefile”. Unfortunately, truly random text is almost useless in this regard, because it doesn’t look anything like a real makefile. Instead, we can use Markov chains to generate random text that is very much like a real makefile. When we first introduced this technique, we uncovered 13 previously unknown incompatibilities — at the time that represented 10% of the total defects reported against the parser! Read on to learn more about Markov chains and how we applied them in practice.

Markov chains

A Markov chain is simply a sequence of random values in which the next value is in some way dependent on the current value, rather than being completely random. Consider the case of generating random text one letter at a time, from the set of uppercase English letters (A-Z). If the sequence is completely random, then for each character generated, any letter is equally probable. Regardless of what characters you’ve generated up to this point, you are just as likely to get a D as an X next. Think of it as if you have a bag of tiles, one for each letter. With truly random text, you pick one tile and write down the letter on that tile. Then you return the tile to the bag and pick again. Lather, rinse, repeat until you’ve generated as much text as you like.

But suppose we said that the probability of the next character is dependent on the last character we generated. For example, in English text if you run across the letter Q you can be pretty sure that the next character is going to be U. It’s almost certainly not going to be X, or Z, etc. We can build a table that tells us the probabilty that any given letter will be followed by any other letter. For the letter Q, the table might look like this:

We can do a much better job of generating text that looks like English if we use these tables to guide us. Imagine that instead of one bag of tiles with one tile for each letter, we have one bag for each letter, and we fill each bag with tiles according to the probabilities in our table. For example, the bag for the letter Q would contain 96 U tiles, and one tile each for A, E, I, and O. Each time we want to generate a new letter, we look at the last letter we generated, find the bag of tiles corresponding to that letter, and pick out one tile. After writing down the letter, we return the tile to the bag and repeat the process. The sequence of letters that we generate in this manner is a simple Markov chain.

How do you build the probability tables? One way is to generate them from some sample input. If we have a sufficiently large example of text in the target language, we can scan it and count the number of times each letter occurs, and the number of times it is followed by each letter. Note that it is critical to have a large and varied input. If the sample is too small, the resulting probability tables won’t accurately represent the target language. The generator will only be able to generate the input text itself.

Markov chains of order m

Of course, we don’t have to limit ourselves to considering just the single previous letter. For example, if the previous two characters are ST, then the next character is probably a vowel, or maybe an R. It’s probably not going to be D, or Q, etc. Just as before, we can make a table that tells us the probability that any given pair of letters will be followed by any other letter. And we can keep going, adding more and more of the preceding letters to our formula. The number of previous characters we use is called the order of the chain, so if we use the previous 4 characters, we would say we have a Markov chain of order 4.

The greater the order of the chain, the “smarter” our generator becomes, because it is considering more context when choosing letters. As you can see, the generated text looks more and more like the target language as you increase the order (although after a some point, you get little additional benefit from further increases):

Using Markov chains in testing

In order to use this technique effectively in testing, you need a couple of things besides the generator itself:

1. A large sample input to seed the generator. As noted, the bigger your sample input text, the higher the quality of the probability tables, and therefore the more varied your generated text will be. Since we are trying to generate makefiles, we used several megabytes of makefiles from a variety of open-source projects as the sample text.
2. An automated evaluation mechanism. You have to be able to determine quickly and automatically if a given generated file is processed correctly or not. Of course, correct can mean many different things here. It might be as simple as “does not cause a program crash”. In our case, it means “emake parses this makefile the same way that gmake does”, so we use gmake as a reference implementation. Note that it doesn’t matter if the generated text truly is a completely valid makefile. In fact most of the time it will not be. What matters is those cases is that emake and gmake both report the same error.

We wrote a simple shell script to drive the testing process. First, it uses the generator to produce several random makefiles. Then it runs each makefile through both gmake and emake, and compares the results. Any differences are reported for further investigation.

Conclusion

Verifying the implementation of an emulator for a complex system is hard, especially when the original system has no formal specification. Using randomly generated input is a useful way to extend the breadth of your testing, and Markov chains make it possible to generate even more useful random input. Our original implementation of this technique uncovered several previously unknown defects, and it continues to pay dividends both by uncovering new defects and by providing a confidence measure for our emulation.

If you want to play with Markov chains yourself, you can download the source for the generator used in this article. NB: the program has only been compiled and used on Linux; on other platforms your mileage may vary. For more information about Markov chains, I recommend Section 15.3 of the excellent book Programming Pearls by Jon Bentley.

If you enjoyed this article about testing techniques, you may enjoy this related article:

• Delta: the coolest tool you’ve never heard of

But this convience comes at a price. Although some of the built-in rules are still relevant in modern environments, many are obsolete or uncommonly used at best. When’s the last time you compiled Pascal code, or used SCCS or RCS as your version control system? And yet every time you run a build, gmake must check every source file against each of these rules, on the off chance that one of them might apply. A simple tweak to your GNU Make command-line is all it takes to get a performance improvement of up to 30% out of your makefiles. Don’t believe me? Read on.

Let’s look at a trivial example:

all: input
	@echo done

Touch the file input, then run gmake with the -d option, so you can see as gmake tries each of the built-in rules. GMake will ramble on for hundreds of lines, as you’ll see. Here’s a sample of that output:

Considering target file `all'.
 File `all' does not exist.
  Considering target file `input'.
   Looking for an implicit rule for `input'.
   ... many lines omitted ... 
   Trying pattern rule with stem `input'.
   Trying implicit prerequisite `RCS/input,v'.
   Trying pattern rule with stem `input'.
   Trying implicit prerequisite `RCS/input'.
   Trying pattern rule with stem `input'.
   Trying implicit prerequisite `s.input'.
   Trying pattern rule with stem `input'.
   Trying implicit prerequisite `SCCS/s.input'.
   Trying pattern rule with stem `input'.
   ... hundreds more lines omitted ...
  No implicit rule found for `input'.
  Finished prerequisites of target file `input'.

What’s going on here? Well, we didn’t provide a rule describing how to build the file input, so gmake is checking to see if any of the built-in rules could be used to generate it. Of course none of them do, so this is all wasted effort. Lucky for us, a single command-line option is all you need to tell gmake not to bother with the default built-in rules: -r. Try that trivial makefile again, this time with -d -r:

Considering target file `all'.
 File `all' does not exist.
  Considering target file `input'.
   Looking for an implicit rule for `input'.
   No implicit rule found for `input'.
   Finished prerequisites of target file `input'.

All the extra nonsense is gone! And even on this toy example, there is a measurable performance improvement: originally, this makefile runs in about 0.015s (average over three runs); with the built-in rules disabled, it’s just 0.012s. But I can see you won’t be convinced by such a trivial example. So let’s try something a bit bigger:

SOURCES:=$(wildcard sub/*.x)
TARGETS:=$(SOURCES:.x=.o)
all: $(TARGETS)
        @echo done

%.o: %.x
        @echo $@

The directory sub contains 15,000 files named 00001.x through 15000.x. With the built-in rules (and redirecting output to /dev/null), this makefile runs in about 60.2s; without the built-in rules, 42.9s — 28% faster.

Finally, let’s try this optimization on a real build. I built one component of the Accelerator project completely, then ran “no-op” builds (ie, no work to be done, just checking that everything is up-to-date). With built-in rules, this took 6.0s; without, it took 5.2s — 13% faster:

Now, if your build actually relies on built-in rules obviously you can’t simply disable them. But you could explicitly define just those rules that you require and disable the rest. For example, if you use the default %.o: %.cpp rule, you could add just that rule to your makefiles:

%.o: %.cpp
	$(COMPILE.cpp) $(OUTPUT_OPTION) $<

Once you’ve done that, you can add -r to your command-line and enjoy the benefits. If you go this route, you can see the list of built-in rules by running gmake -p; the built-ins are marked as “built-in” in that output.

Conclusion

Disabling gmake’s array of built-in rules is an easy way to squeeze extra performance out of your makefiles, particularly on large builds and on no-op builds. All you have to do is add -r to your commmand-line. (NB: If you prefer more descriptive command-lines, you can use –no-builtin-rules instead!)

• Makefile performance: recursive make
• Makefile performance: $(shell)
• Makefile performance: pattern-specific variables
• Makefile performance: built-in rules (this post)

First, let me give a simple demonstration. Here’s the makefile:

16:=x x x x x x x x x x x x x x x x
input_int:=$(foreach a,$(16),$(foreach b,$(16),$(foreach c,$(16),$(16))))
decode=$(words $1)
encode=$(wordlist 1,$1,$(input_int))
decr=$(wordlist 2,$(words $1),$1)
decr2=$(wordlist 3,$(words $1),$1)
eq=$(filter $(words $1),$(words $2))
g0:=
g1:=x

fib=$(if $(filter-out undefined,$(origin f$1)),\
           $(f$1),\
           $(if $(call eq,$1,$(g0)),\
	          $(eval f$1:=$(g0))$(g0),\
                  $(if $(call eq,$1,$(g1)),\
                         $(eval f$1:=$(g1))$(g1),\
                         $(eval f$1:=$(call fib,$(call decr2,$1)) $(call fib,$(call decr,$1)))$(f$1))))

print=$(if $1,\
$(call print,$(call decr,$1))$(info $(call decode,$1): $(call decode,$(f$1))),\
$(info 0: 0))

%:
	@:$(if x$(call fib,$(call encode,$@)),$(call print,$(call encode,$@)),)

Invoke it with a single argument, the length of the sequence to generate:

ericm@chester:~/blog/fibonacci$ gmake 10
0: 0
1: 1
2: 1
3: 2
4: 3
5: 5
6: 8
7: 13
8: 21
9: 34
10: 55

Nifty!

Now, although this demonstration is not especially practical, it does make use of a few advanced gmake concepts: arithmetic; caching dynamically generated variables; and recursive functions.

Arithmetic

None of this would be possible without support for arithmetic operations. Here’s a brief explanation of how this works (for more details, the Mr. Make article Learning GNU Make Functions with Arithmetic): we use strings of space-separated x characters to represent values; for example, the number five is represented as x x x x x. To add numbers together, we concatenate the string representations, and to subtract, we trim the appropriate number of x‘s from the string. To convert from the string representation to the numeric value, we just count the number of x‘s in the string, using the $(words) builtin, and finally, to convert from the numeric value to the string representation we extract the appropriate number of x‘s from a canonical string that just has a series of several thousand x‘s.

This is inefficient, in both memory and time. The representation of a number requires double the value in bytes, so the number 50,000 requires 100,000 bytes of storage. Converting to a numeric value from the string representation is a linear operation that scales with the magnitude of the value — the bigger the value, the longer the conversion takes. These factors together limit the range of numbers that you can practically work with using this scheme, although it works fine for small values (up to around 10,000 or so). For the Fibonacci makefile, the inefficiencies mean that we can only generate the sequence out to about the 40th value. At that point, gmake has already sucked up 1.4 GB of memory!

Caching dynamic values

Because it is time-consuming to compute Fibonacci numbers this way, we’d like to cache the results, so we don’t ever duplicate work. But the values are dynamically generated. For that matter, the variables themselves must be dynamically generated — we don’t know beforehand which Fibonacci numbers we’re going to have to compute. So how do you dynamically generate variables and cache their values in gmake? With $(eval). You can read all about it in another Mr. Make article, $(eval) and macro caching. In our Fibonacci makefile, you can see that as we determine each Fibonacci number, we invoke $(eval) to save the value. The fib function is then setup to first check for the existence of a cached value and only bother with the computation if there is no cache entry.

Recursive functions

The Fibonacci sequence is defined recursively as f(n) = f(n – 1) + f(n – 2), so naturally we use a recursive function in our implementation. The fib function takes one argument, the index of the Fibonacci number to compute, encoded using the scheme described above. After checking the cache, fib checks if the current index is either zero or one. By convention, these Fibonacci numbers are defined to have value zero and one, respectively, so there is no need to compute them. This also serves as a terminating condition so we don’t recurse infinitely.

If the current index is neither zero nor one, then fib calls itself recursively twice, to compute the two preceding Fibonacci values. The recursive results are combined, cached, and returned as the overall result of the fib function itself.

Conclusion

I’m surprised at how quickly gmake computes the sequence, actually. On my laptop, gmake computes the first 30 values in a fraction of a second. After that the inefficiencies in the arithmetic operations really come into play: it takes about 3.5 seconds to compute 35 values, and 25 seconds to compute 39 values. Still, considering the limitations of the environment, I think it’s impressive that it works at all.

Here’s a question for you: if ElectricAccelerator had an extension that allowed you to explicitly mark a non-pattern rule as having multiple outputs, would you use it? For example:

#pragma multi
something otherthing: input
	@echo Generating something and otherthing from input...

What do you think? Comments encouraged.

In an earlier post I compared the performance of ElectricAcclerator and distcc by building samba using each tool in turn on the same cluster. In that test I found that Accelerator bested distcc at suitably high levels of parallelism, but that distcc narrowly beat Accelerator at lower levels of parallelism. At the time I chalked the difference up to slightly higher overhead associated with Accelerator. But you must have known I couldn’t just leave it at that. I had to know where the overhead was coming from, and eliminate it, if possible. The exciting conclusion is after the break.

Samba Reloaded

To recap, I previously found that samba is a very CPU-intensive build, despite being written entirely in C. This fact was demonstrated empirically by examining the performance on one dual-core host using just gmake -j at varying levels of parallelism. Past -j 2, the performance degraded sharply. For the distcc and emake tests, I used a cluster of 12 dual-core hosts. Eleven served as workers, for a total of 22 CPU’s. The remaining host was used as the build host (and cluster manager, for emake tests). Here are the original results:

Until we got to about 11 CPU’s, distcc appeared to have a slight edge on emake. I had lots of theories that could explain the difference: maybe our Electric File System (EFS) was slower than ext3fs, or maybe the Electric Agent was sluggish in supplying metadata to the EFS, or in processing file usage data from it. Maybe lock contention in emake itself was causing the problem.

Of course before I could test any of these theories I had to make sure I could reproduce the original behavoir. I set up a five-node cluster using the same dual-core hosts I used previously, plus one additional node to serve as the build host (unfortunately the other half of the cluster was reserved for other tests — I’m not the only person doing work here at Electric Cloud, after all!). This gave me a total of 10 worker CPU’s. After installing the latest version of Accelerator (4.5.0), I fired off a series of three builds each with Accelerator and distcc, using 10 workers. When those builds completed, I computed the average build time for each tool — and found that Accelerator beat distcc by a small margin.

Deeper Into the Rabbit Hole

This result was wholly unexpected, given the results from the previous tests. The next step was to run a series of builds with varying numbers of workers, from 1 to 10. Here are the results:

Now the results are more in line with my expecation: with low levels of parallelism distcc appears to perform better, but Accelerator catches up and finally surpasses distcc once enough resources are engaged. The breakeven point has moved though, from about 11 CPU’s to about 9 CPU’s. In addition, there was an outlyer in both sets of results: with just one worker, Accelerator was consistently faster than distcc. That didn’t fit well with my theories — if the EFS was slow, for example, then Accelerator would have been slower than distcc with one worker.

A New Theory

As I puzzled over this new data, something clicked that caused me to remember a subtle difference between distcc and Accelerator: they use different strategies to determine how to allocate jobs to workers in the cluster. Accelerator prefers to fully load one host before running jobs on another host; distcc prefers to spread the load across as many hosts as possible before doubling up on any one host. The following images illustrate the result obtained when using these different strategies to assign five parallel jobs to a cluster of ten workers on five hosts:

This realization led to a new theory: perhaps the performance difference observed with low numbers of workers was simply an artifact of this difference in worker allocation strategies. We’ve already seen that this particular build is especially CPU intensive. Two jobs on one host have just two CPU’s that they must share; two jobs on two hosts have four total CPU’s available. This theory would also explain why emake “catches up” with distcc — as more and more jobs are run in parallel, distcc is forced to assign multiple jobs to a single host. Eventually, both systems have fully loaded all the available cluster nodes, so the difference in allocation strategies becomes moot.

Armed with this theory, I altered my benchmark so that emake would use the same allocation strategy as distcc, by explicitly enabling and disabling agents via the cluster manager. For example, for a trial with two agents, I enabled one agent each on two cluster nodes. This technique allowed me to better compare the relative performance of distcc and Accelerator. Here are the results from this test:

With the allocation strategy out of the equation, Accelerator actually has a small, consistent edge over distcc (about 2%) on small clusters. And the previous test showed that Accelerator scales better than distcc, so on large clusters the difference is even more pronounced (about 15%).

Should We Change Accelerator?

An obvious question is whether we would consider changing the allocation strategy in Accelerator. The answer is probably no. The strategy we use, although suboptimal for this particular build, actually works very well across a wide variety of builds. One of the key advantages of this strategy is that it allows Accelerator to minimize network overhead, since agents on a single host can share various kinds of data directly. There are relatively few builds that skew so heavily towards CPU utilization, so changing the strategy to benefit those special cases at the expense of the more common case seems unwise.

A Champion Vindicated

Although we previously declared Accelerator the victor versus distcc when building samba, it was not without some reservations. With the new results shown here, I’m satisfied that we made the correct decision: CPU-for-CPU, Accelerator is more efficient and scales better than distcc, at all cluster sizes.

Pattern-specific Variables 101

Pattern-specific variables are similar to target-specific variables in that they define a variable value that applies only in a particular context, but where target-specific variables apply to a single specific target, pattern-specific variables apply to all targets that match the given pattern. This gives us a way to get target-specific-variable-like behavior when using pattern rules. It also allows us a more flexible syntax for target-specific variables, even in the absence of pattern rules. A few examples should make everything clear. First, regular target-specific variables:

.PHONY: a b
FLAGS=default flags
all: a b

a: FLAGS=special flags for a
a b:
	@echo $@ using FLAGS='"$(FLAGS)"'

Here we have defined FLAGS with a default value at the global scope, and a custom value for target a. When you run this makefile with gmake, you’ll see the value used for FLAGS is different for the two targets, as expected. Now, let’s look at typical pattern-specific variable usage:

.PHONY: a b
FLAGS=default flags
all: a.x b.x c.x d.z e.z

%.x: FLAGS=special flags for .x files
%.x:
	@echo $@ using FLAGS='"$(FLAGS)"'

%.z:
	@echo $@ using FLAGS='"$(FLAGS)"'

Again we have a custom value for FLAGS, but this time it applies to all targets matching the pattern %.x. We could achieve the same behavior by using normal target-specific variables for each of the .x files of course, but the pattern-specific variable definition is more succinct and convenient, and it is more robust, since we need not hardcode the list of targets to which the new definition applies. Finally, let’s look at pattern-specific variables without pattern rules:

.PHONY: foobar foobaz fooboo booboo
FLAGS=default flags
all: foobar foobaz fooboo booboo

foo%: FLAGS=special flags for foo files

foobar:
	@echo Building foobar using FLAGS='"$(FLAGS)"'

foobaz:
	@echo Building foobaz using FLAGS='"$(FLAGS)"'

fooboo:
	@echo Building fooboo using FLAGS='"$(FLAGS)"'

booboo:
	@echo Building booboo using FLAGS='"$(FLAGS)"'

With this example you can see that pattern-specific variables are used for each of the foo files, even though we have specified explicit rules for each of those files. That implies that gmake searches through the pattern-specific variables looking for variables that should apply for a given target independent of the search for a rule to build the target.

What happens when multiple patterns match my target?

Suppose we modify our previous makefile like this:

.PHONY: foobar foobaz fooboo booboo
FLAGS=base
all: foobar foobaz fooboo booboo

foo%: FLAGS+=extra_foo_flags
%boo: FLAGS+=extra_boo_flags

foobar:
	@echo Building foobar using FLAGS='"$(FLAGS)"'

foobaz:
	@echo Building foobaz using FLAGS='"$(FLAGS)"'

fooboo:
	@echo Building fooboo using FLAGS='"$(FLAGS)"'

booboo:
	@echo Building booboo using FLAGS='"$(FLAGS)"'

What would you expect the value of FLAGS to be when building fooboo? After all, both foo% and %boo match the target fooboo. In fact, there are two possibilities, both perfectly reasonable, both consistent with at least some other aspect of gmake behavior.

The first possibility is that when building fooboo, FLAGS has the value base extra_foo_flags. That is, gmake applies the pattern-specific variables from the first, and only the first, pattern that matches the target. This is consistent with the way that gmake searches patterns to find a rule to build a target: as soon as one match is found, gmake stops searching. GNU make 3.80 uses the “first match” policy.

The second possibility is that FLAGS has the value base extra_foo_flags extra_boo_flags when building fooboo. That is, gmake applies all the pattern-specific variables from all patterns that match the target, in the order the variables are defined in the makefile. This is a bit more intuitive, and is more consistent with the way variable definition in general works in gmake. GNU make 3.81 uses the “all matches” policy.

Performance Comparison

In both 3.80 and 3.81, the search for pattern-specific variables that apply to a given target involves a search of the pattern-specific variable definitions. The difference is that in 3.80, the search scales with the number of patterns that have pattern-specific variable definitions, and the search can stop as soon as a match is found. In gmake 3.81, the search search scales with the number of pattern-specific variable definitions, and the search must always inspect every single definition, even after the first match is found. To demonstrate the impact of this change, I did a series of tests. I created several makefiles like the following:

%: FOO:=abc
%: FOO:=abc
%: FOO:=abc
...
all: 1 2 3 4 5 ... 10000
1 2 3 4 5 ... 10000: ; @/bin/true

These makefiles each have a single all target with 10,000 prerequisites. I varied the number of pattern-specific variable definitions from 1,000 to 80,000 and timed how long it took to run the makefile in dry-run mode with both gmake 3.80 and 3.81. For kicks, I also included the runtimes for ElectricAccelerator (emake) in 3.81 emulation mode. Here are the results:

No doubt some of you are thinking that this is a toy example, so not particularly applicable to the real world. After all, nobody assigns the same variable over and over like that. That’s an excellent point. Unfortunately, when I tried to create 20,000 unique pattern-specific variables, gmake 3.81 crashed after sucking up all the available RAM on my system. Oops!

The bigger question is, who would ever have tens of thousands of pattern-specific variables? The answer is: people who have switched from a recursive build to a non-recursive build. In fact, I have seen a single makefile with over 180,000 pattern-specific variables, attached to just 18 distinct patterns. The point is, as crazy as it seems, builds of this scale do exist in the “real world”.

Are you at risk?

To find the pattern-specific variable definitions in your makefiles, you can use the following command:

egrep "^[^:]*%[^:]*: *[^=]*=" Makefile

If you have lots of pattern-specific variables, what can you do to reduce the performance impact? A few ideas come to mind:

• Switch to gmake 3.80 and hope that a future release will address this problem.
• Convert your pattern-specific rules to explicit target-specific rules, perhaps using $(eval) to generate the new variable definitions dynamically so you don’t have to type out every one by hand.
• Switch to a recursive build, which would allow you to partition your pattern-specific variables so that gmake only ends up searching the variables that are likely to apply to the targets referenced in a given makefile.
• Use ElectricAccelerator, which emulates gmake 3.81 behavior but uses a more efficient algorithm in its implementation.

Update: Restored the results graph to the post, which mysteriously disappeared thanks to WordPress.

• Makefile performance: recursive make
• Makefile performance: $(shell)
• Makefile performance: pattern-specific variables (this post)
• Makefile performance: built-in rules