What does annocat do?

Annocat has a single purpose: concatenate a series of annotation files from real emake invocations into one annotation file representing a single logical build. In order to do this correctly we have to do a few transformations on the original data:

• Job identifiers from each source file are rewritten so they are scoped by the build identifier of the build containing that job. For example, if we have a job with identifier J0830fdc0 in build 12345, annocat will replace that identifier with J12345_0830fdc0, not only in the <job> tag but also anywhere else the identifier appears in annotation, such as in the <waitingJobs> tag. This transformation ensures that we don’t have collisions between job identifiers in different source builds.
• Timing information is adjusted so that it is relative to the start of the logical build, rather than relative to the start of any individual actual build.
• Environment and properties blocks are discarded from all but the first real build. The metrics block is discarded entirely.
• Additional pseudo-jobs are created in the result to tie all the real builds together into a single logical build. The logical build represents the actual builds as submakes spawned from a series of serialized jobs in a synthentic make instance.

How do I use annocat?

Annocat works like the standard cat utility, but on annotation files. Download it here, then invoke it like this:

perl annocat.pl build_1234.xml build_1235.xml build_1235.xml > combined.xml

After running annocat, you’ll can load the result in ElectricInsight and run all your favorite reports.

How does annocat work?

Annocat is a simple Perl script that uses the standard Perl streaming XML parser XML::Parser to process the annotation file one tag at a time. I chose the streaming parser because annocat does not need to track a lot of state, so we don’t need the sophistication of a DOM-style parser. I chose to implement annocat in Perl rather than Tcl-and-annolib because I wanted to show how you might work with annotation data in Perl; because you don’t need all the power of annolib for this simple task; and because I wanted to remind myself how much I dislike Perl.

The basic premise of annocat is simple: as each tag is read from a source annotation file, annocat checks the type of the tag and performs any required transformations, then prints the tag to standard out. Although it’s straightforward stuff, the final script is a few hundred lines of code, so I won’t go through it line-by-line here. I will point out a couple of tricky bits, however.

First is the bit where annocat adjusts timing information. A global variable, gElapsed tracks the elapsed time as of the start of the annotation file currently being processed. This variable starts at zero and is updated after each annotation file is completed. You can see the update in the main loop of the program, around line 287. When annocat emits the timing data for a job, around line 153, it just adds the elapsed time to the real timing data extracted from the annotation file, thereby shifting the logical start time by the required amount.

Second is the bit where annocat emits pseudo-jobs to provide the top-level structure of the logical build. The tricky part is determining when to emit these jobs, and making sure that they are emitted in the correct context — that is, in keeping with the annotation format, the rule job that spawns a submake must immediately preceed the opening <make> tag for the submake, and the rule job should list the parse job of the submake as a waitingJob. Since we’re using a streaming parser, by the time we get to that parse job, unfortunately, we will already have processed and emitted the opening <make> tag. So we have to do something a little more clever around the start of each build: instead of blindly copying the first <make> tag for the build to standard out, annocat buffers that tag temporarily, until it gets to the first <job> tag in that build. Then we have the information we need to create the fake rule job, so we do so, and only then do we emit the buffered <make> tag. Switching into buffered mode occurs around line 126, when annocat detects that it has found a new <build> tag; emitting the fake rule job occurs around line 181, when annocat detects the first job in the new build. Around that line you’ll also see where annocat outputs the follow job for the previous build.

Future work

Although it is functional as currently implemented, there’s more that could be done with annocat. First, it would be nice if it didn’t just dump the environment data from the second and subsequent real build. One way to handle it would be to move it under the first make instance in the build and using environment-level annotation format to capture the deltas between the new environment and the environment for the first build. Second, it would be nice if annocat didn’t drop the data in the <metrics> sections. One solution would be to aggregate the metrics from all source builds and emit a single unified block of metrics for the logical build. I leave these enhancements as an exercise for the reader.

I was reminded of this issue when it popped up again on the GNU make-help mailing list. Take a look at this recent post:

Unfortunately, as this poster discovered, there is no really good solution to this problem, at least not unless there is help from the build tool itself. Some people have tried tricks like colorizing the output by piping each command through a filter that sets the font color before printing the output, but any such solution is going to be unreliable because it doesn’t address the fundamental problem of having multiple processes trying to write output simultaneously. Plus it’s a real nuisance to modify your makefiles to add the filter.

How ElectricAccelerator Does Build Logs

Seeing that message on make-help reminded me that I take for granted the way that Accelerator handles the build log. Simply put, Accelerator emits the build log in correct serial order, without any interleaving of output from different commands — even though those commands are running in parallel and frequently out-of-order. The log you get at the end of the build is identical to the log you would have gotten if you’d run a single-threaded, serialized build.

Think about that for a second: the build log is emitted in serial order. No more scrambled build logs. No more kludgy partial solutions, because we solved the problem at the level of the build tool. You just get consistent, predictable, useful logs. Every time.

Consider this trivial makefile:

  all: a b c

  a b c:
          @for n in 1 2 3 4 ; do echo $@-$$n && sleep 1 ; done

Now compare the output when run by regular old serial gmake with the output when run by parallel gmake -j 4, and output from Accelerator (emake) (red text indicates places where the output differs from serial gmake, bold text indicates places where the output differs from the previous run of the same make variant):

The parallel gmake output is different from serial gmake on every run, and what’s worse, the parallel runs are not even consistent with each other from one run to the next! On the other hand, the output from Accelerator is identical to serial gmake, every time.

Anything we can do, we can do better

Of course we’re never satisfied with simply solving a problem that plagues users around the world. We want to provide a solution that is so far superior to the alternatives that you’d be crazy not to want it. Therefore, we created annotated build logs, or simply annotation, in late 2003. Annotation is a version of your build log marked up with XML to provide bundles of information above and beyond the standard build output log, in a format that is easily searched and manipulated. This is the solution that everybody wishes they had, and the solution that the developers of other parallel make tools wish they had thought of first. Here’s a bit of the annotation for our example build:

  <job file="Makefile" name="a" neededby="J01513660" id="J01511fb0">
  <argv>for n in 1 2 3 4 ; do echo a-$n && sleep 1 ; done
  </argv>
  <output src="prog">a-1
  a-2
  a-3
  a-4
  </output>
  </job>
  <job file="Makefile" name="b" neededby="J01513660" id="J01511fb8">
  <argv>for n in 1 2 3 4 ; do echo b-$n && sleep 1 ; done
  </argv>
  <output src="prog">b-1
  b-2
  b-3
  b-4
  </output>
  </job>

There are a few things in particular that I want to point out in this snippet:

1. Every job in the build is reported separately from every other job, with the commands and output from each easily identifiable.
2. You can generate the traditional build output log from annotation simply by concatenating the text in the <output> tags.
3. Annotation includes even those commands that are not echoed to the standard build output log because of the use of the “silent” prefix (@) in the Makefile!

If you are comfortable with XML you can probably already see the benefits here: you could easily put together some scripts with Perl or even just with xmlgrep and xmlfmt that search the content of <output> tags for things that look like warnings or errors, then reports precisely which job produced that output. Alternatively, you can use our annolib library to build that script:

#!/usr/bin/tclsh

load annolib.so

set anno [anno create]
set xml  [open emake.xml r]
$anno load $xml
close $xml

$anno jobiterbegin
while { [$anno jobitermore] } {
    set job [$anno jobiternext]
    set match false
    foreach commandBlock [$anno job commands $job] {
        foreach {lineNumber argv outputs} $commandBlock {
            foreach {src text} $outputs {
                if { [regexp -nocase {(warning)|(error)} $text] } {
                    set match true
                }
            }
        }
    }
    if { $match } {
        puts "Problem found in job $job ([$anno job name $job]):"
        foreach commandBlock [$anno job commands $job] {
            foreach {lineNumber argv outputs} $commandBlock {
                puts "\t$argv"
                foreach {src text} $outputs {
                    if { $src == "prog" } {
                        puts "\t$text"
                    }
                }
            }
        }
    }
}

Just like that, you have a simple script you can use as a starting point for your own post-build reporting on errors and warnings in your build.

So with ElectricAccelerator, you get the best of both worlds: fast parallel builds, and build logs that you can actually use. Just another way that Accelerator rocks.