To evaluate Accelerator cache efficiency, we need to compare the total amount of file accesses performed by the build with the amount that actually ends up hitting the host filesystem. The difference between these two values tells us how much of the total is served from the cache; in turn, the ratio between this difference and the total gives us the cache hit rate. For this comparison, we’ll look at the following specific types of filesystem access:

• The number of readdir() operations.
• The number of stat() operations.
• The amount of data read from disk.
• The amount of data written to disk.

Counting total file accesses

We can get the total number of accesses from agent performance metrics. We’ve looked at these once or twice before. Follow the directions here to obtain and aggregate metrics from the agents (make sure to get the latest version of the agentsummary script). Once you have that summary data, the Directory scans “to EFS” metric in the Caching section gives you the total number of readdir() operations performed by the build:

Caching:
  ...
  Directory scans:  15.0% (40 to EFS, 6 to emake)
  ...

Next, you need the raw count of Lookup records from the Usage records section; this is the total number of stat() operations performed by the build:

Usage records:
  ...
  Lookup              4029 ( 12.7%),   21.3 per job;   125881 raw (  3.2%)

Finally, you need the Total MB values for EFS disk reads and EFS disk writes from the Bandwidth section; these give us the total amount of data read and written by the build:

Bandwidth:
  ...
  EFS disk reads
    Locked:               295.4 MB, 223.6 MB/s active,   7.4 MB/s overall
    Unlocked:               0.0 MB,   0.0 MB/s active,   0.0 MB/s overall
    Total:                295.4 MB, 223.6 MB/s active,   7.4 MB/s overall
  EFS disk writes
    Locked:               218.9 MB, 280.6 MB/s active,   5.5 MB/s overall
    Unlocked:               0.0 MB,   0.0 MB/s active,   0.0 MB/s overall
    Total:                218.9 MB, 280.6 MB/s active,   5.5 MB/s overall
  ...

Counting host filesystem accesses

Now we need to find out how much of the I/O activity actually hit the host filesystem — all the accesses that were not serviced by the cache. You’ll find this data in the emake performance metrics, which we’ve looked at a bit previously. To obtain these metrics, you need to enable emake performance logging by adding --emake-debug=g --emake-logfile=emake.dlog to your emake command-line options. When your build completes, you’ll find the metrics in the file emake.dlog. First, we need the DirCache readdirs and DirCache stats values in the Counter values section; these give us the number of readdir() and stat() operations that hit the host filesystem, respectively:

Counter values:
  ...
  DirCache readdirs         10
  DirCache stats           890
  ...

Next, we need the To disk and From disk data from the Bandwidth section; these tell us the amount of data written to and read from the host filesystem:

Bandwidth:
 ...
 To disk:             167.6 MB,  28.6 MB/s active, 4.1 MB/s overall
 From disk:            29.1 MB,   6.2 MB/s active, 0.7 MB/s overall
 ...

Computing cache efficiency

Now we can put all the numbers together and compute the cache efficiency:

The effect is dramatic, even on the small build I used for this example. This is why some people call emake a “ClearCase accelerator”. Of course, every build will have a slightly different profile, but in general you ought to see similar results.

Usage records

As a build is executed by Accelerator, the Electric File System tracks all filesystem and registry accesses made by commands in the build. This usage log is the means by which modifications made on cluster nodes are propagated back to the emake host, and it also forms the basis for our conflict detection algorithm. Because it is such a core piece of the system, we track numerous metrics specifically related to the usage log. Those metrics are reported in the Usage records section of the agent performance metrics.

After your build completes, use cmtool and agentsummary to get the agent metrics, then find the Usage record section. It looks something like this:

Usage records:
  Append                 0 (  0.0%),    0.0 per job;       25 raw (  0.0%)
  Blind create           0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Blind truncate         0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Create              4835 (  2.5%),    3.7 per job;     5425 raw ( 89.1%)
  Create key             0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Create(dir)          380 (  0.2%),    0.3 per job;      476 raw ( 79.8%)
  Delete key             0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Delete value           0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Failed lookup     112444 ( 59.3%),   85.8 per job;        0 raw (  0.0%)
  Link                 760 (  0.4%),    0.6 per job;        0 raw (  0.0%)
  Lookup             19715 ( 10.4%),   15.0 per job;  7507140 raw (  0.3%)
  Lookup key          1426 (  0.8%),    1.1 per job;     8385 raw ( 17.0%)
  Modify                33 (  0.0%),    0.0 per job;       42 raw ( 78.6%)
  Modify atts            1 (  0.0%),    0.0 per job;     1714 raw (  0.1%)
  New name            4080 (  2.2%),    3.1 per job;   116978 raw (  3.5%)
  Read               43476 ( 22.9%),   33.2 per job;    85315 raw ( 51.0%)
  Read key               0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Read value          2274 (  1.2%),    1.7 per job;     2274 raw (100.0%)
  Rename                 0 (  0.0%),    0.0 per job;       11 raw (  0.0%)
  Rename(dir)            0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Set value              0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Set value if           0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Truncate               7 (  0.0%),    0.0 per job;       32 raw ( 21.9%)
  Unlink                 0 (  0.0%),    0.0 per job;        0 raw (  0.0%)
  Unlink(last)         225 (  0.1%),    0.2 per job;      910 raw ( 24.7%)
  Total             189656 (100.0%),  144.7 per job;  7728727 raw (100.0%)

As you can see, Accelerator tracks a wide variety of file and registry operations:

For each type of operation, the agent reports the following data:

• Total number of records of that type reported to emake.
• Portion of the total usage records reported represented by that type.
• Average number of records of that type per job in the build.
• Total number of raw records of that type reported by the EFS.
• Portion of the raw records of that type reported to emake by the agent.

Why are there so many types of operations?

As of Accelerator 4.3.0, the EFS and agent track 25 different types of operations, almost double the variety that we tracked in Accelerator 3.5.0. Many of the different types are fairly subtle variations on one another. For example, the distinction between Create and Blind create is simply the flags used in the system call that created the file. Similarly, Create could be considered a special case of Modify. The question inevitably arises: wouldn’t it be simpler to track fewer types of operations?

The answer of course is yes, it would be simpler, but that simplicity would come at the cost of superior efficiency. For example, although Create is a special case of Modify usage, it has one very significant difference: Modify usage implies that a command used the previous contents of the file before making changes, but Create implies that the command did not use the previous contents. This seemingly trivial distinction is the difference between commands that must be serialized and commands that can run in parallel: two commands that modify the same file must be serialized (think of compile steps that each update a shared .PDB file), while two commands that each overwrite the same file can be run in parallel.

Reported Usage versus Raw Usage

You may have noted that there is sometimes a significant difference between the raw usage reported by the EFS to the agent, and the usage ultimately reported by the agent to emake. The simple explanation is that the agent has some smarts to eliminate redundant usage records, in order to minimize the number of records emake must manage and examine for conflicts.

A trivial example is eliminating redundant Read operations from the usage: the first time a file is read during a given job, the agent logs the usage. The second time the file is read during the same job, the agent discards the usage because it is redundant with the usage already logged for the file.

A more sophisticated example is eliminating Read operations on new files: when a file is created during a job, the agent logs the usage. If the file is later read during the same job, the agent discards the usage — the read cannot impact the dependency analysis that emake will perform, since it is reading data that was generated in the same job, rather than data generated during a different job in the build.

By volume, the most significant application of this technique is in the reduction of lookup operations. It’s astonishing just how many lookup operations occur in the course of an average build, and even more so how many of them are redundant. In the sample shown above, only a tiny fraction of the lookups were actually significant. The rest were discarded by the agent before ever being sent to emake, dramatically reducing the amount of data emake must manage. If anything, the sample shown here is on the low end in terms of the number of lookups generated — most builds generate far more. One customer’s build generated about 200 million lookups during a 50 minute long build.

Some types of usage, such as failed lookups, are not actually directly reported by the EFS. Instead, these types of records are synthesized by the agent to replace longer sequences of operations — again, in order to reduce the amount of data that emake must track.

What’s normal?

The usage profile of most builds is surprisingly similar: the vast majority of usage is lookups (both failed and successful); reads are the next most common, followed by creates. Just about everything else occurs so infrequently that it’s not worth calling out explicitly. To put concrete values on it:

How can I use this data?

There are two ways you can use the usage metrics to improve build performance. First, you can use the usage profile as a simple “gut check” to spot anomalous behavior. For example, a build that logs significantly more Create usage than others may be doing unnecessary additional work, such as building targets repeatedly. A build that logs excessive Read usage may indicate problems like badly factored header files which cause every compile to read every header. It’s hard to predict exactly what the problems might be; the key is to keep your eyes open for anything out of the ordinary.

The second consideration relates specifically to failed lookup usage. As you’ve seen, failed lookups account for the majority of all usage. Earlier I noted that failed lookups correspond to stat()-like operations on non-existent files, but what does that really mean?

The most common source of failed lookups is include path searches. Suppose your build specifies an include search path to the compiler, for example:

... -I/tools/megawidgets/include -I/tools/apache/include \
  -I/tools/expat/include -I/tools/tcl/include -I/tools/perl/include ...

Now, imagine your source files contain #include directives like this:

#include "math.h"

When the compile looks for the file math.h, it will search for it in each directory specified in the include search path. Of course, the file exists in only one directory, so each attempt to find it in the other directories generates failed lookup usage. Now imagine a system in which the include search path contains dozens of directories. Every compile step in the build will search each of those directories for each header file included. You can see how quickly it adds up:

      500 compiles
  *    20 headers per compile
  *    50 dirs in include path
  ----------------------------
  500,000 failed lookups

Depending on the build, there may be nothing you can do about this. But if you are able to, you could investigate optimizing the include search path:

• Eliminate stale entries.
• Reorder the include path so that more commonly used headers are found earlier.
• Customize the include path per target. Some builds use a single include path for all compile steps, but if the megawidgets headers (for example) are used only by some targets, adding that directory to the include path for the other targets only bloats the search space.

Unfortunately, it’s hard to predict exactly how much impact changes like these could have on your build, although I can say that my experience has consistently shown that reducing the size of your build in any dimension often results in surprising performance benefits due to second-order effects. I encourage you to go for the “low hanging fruit” — make those changes that are easy to make and measure the impact. If the modifications pay off, find the next lowest hanging fruit and repeat the process. Keep going as long as the cost of making the changes is less than the benefit you see.

Understanding the performance of parallel, distributed systems can be difficult. Fortunately, Accelerator provides a wealth of data to facilitate performance analysis. In this first continuation of my presentation at the 2008 Customer Summit, I’ll show you how to collect agent performance metrics, and how you can use them to get a quick overview of the performance characteristics of your build.

Collecting Agent Metrics

Every agent in the cluster automatically tracks performance metrics for every build that it participates in, so you don’t have to do anything special to enable them. All you need to do is download them from the agents after your build has finished. Note that by default, the agent keeps performance data for the previous 100 builds that it participated in, so you don’t have to grab the metrics immediately after your build completes.

The easiest way to get the metrics from the cluster is via cmtool:

cmtool --cm=... runAgentCmd "session performance buildId " > buildId.agentraw

This will write a dump of performance metrics from each agent that participated in the specified build to the file named buildId.agentraw. Next, you’ll want to use the (unsupported) agentsummary utility to produce an aggregate summary of the metrics across all agents:

tclsh agentsummary buildId.agentraw > buildId.agentsum

Now we have something that we can dig into.

Overall time usage

The first thing to look at in the agent performance metrics is the “Overall time usage” section. This divides the time used by the agents into several coarse-grained buckets:

Overall time usage:
  Startup:           4.65s ( 0.0%)       20 intvls, avg   232.5ms
  Cmd setup:       253.15s ( 0.4%)    42281 intvls, avg     6.0ms
  Command:       36479.41s (61.7%)    82823 intvls, avg   440.5ms
  Emake request:  7732.26s (13.1%)   763879 intvls, avg    10.1ms
  A2A request:    1968.67s ( 3.3%)    52112 intvls, avg    37.8ms
  Command end:     448.59s ( 0.8%)    82823 intvls, avg     5.4ms
  Return:         4874.27s ( 8.2%)    42281 intvls, avg   115.3ms
  Idle:           6726.62s (11.4%)    42261 intvls, avg   159.2ms
  End:             671.13s ( 1.1%)

These timers correspond to the following activities:

For each timer, the metrics show:

• The total time spent in the timer across all agents;
• The portion of time spent in the timer as a percentage of the total time across all agents;
• The number of times the timer was active;
• The average amount of time the timer was active each time it was in use.

Note that not all timers are available in all versions of Accelerator.

Of these timers, Command is perhaps the most interesting, because it represents the actual work performed during your build. The other timers all correspond to inefficiency of some sort, whether due to the overhead introduced by running a distributed build across the network, or due to serializations in the architecture of the build itself.

One interesting factoid about the Command timer is that you can use its percentage to estimate the X factor, or speedup relative to a serial build. Simply multiply the Command percentage by the number of agents that participated in your build. For example, if the Command timer represents 65% of the total time, and your build used 8 agents, then the X factor is about 5.2x:

0.65 * 8 == 5.2x better than serial

Two caveats about this value: first, it’s only an estimate. In my experience it seems to be reasonably accurate in the back-of-the-envelope sense, but it’s by no means perfect. Second, it only really works if the agents are running on hardware comparable to the system you would have used for a serial build. If the agents are slower or faster than the baseline host, that will skew the number correspondingly.

Are my values “good”?

Of course the whole point of looking at these timers is to determine if your build is performing well, or if there is room for improvement. Therefore, we need to establish some guidelines for evaluating the values in these timers. To that end, here are some very rough guides:

Remember, Command time represents all the time spent doing the actual work in your build. Ideally, we’d like to see that at 100%, but of course that target is unattainable — if nothing else, there is always overhead introduced simply because we have to move data across the network. Still, it’s not uncommon to see builds with Command time representing 80% or more of the total time. Anything over 60% is pretty good, really. Less than 50% means there’s probably room for improvement. That’s not a guarantee that you can do better, of course. It may simply be the nature of the beast, or you may find that the cost of improving the performance outweighs the benefit. If you do see a low percentage for Command time, the thing to do is look at the other timers to determine where the time is going instead. That will in turn provide some guidance to the root cause of the problem.

For example, if the Emake request timer is high (greater than 15%), that indicates a problem with the communication between emake and the agents. The things to check next in this case are:

• Network bandwidth between emake and the agents.
• System load on the emake host.
• Disk performance on the emake host.
• Number of agents participating in the build (too many agents can cause emake to thrash).

Another possibility is that the Idle and End timers are high (more than 20% combined). This usually indicates that your build has significant serializations, or that it had a large number of conflicts. Use ElectricInsight to investigate further.

Finally, if the Return timer is high (greater than 20%), you’ll want to check:

• Network bandwidth between emake and the agents.
• Disk performance on the emake host.
• Disk performance on the agent host.

It’s important to note that in general, any one metric will not exactly pinpoint a specific problem; rather, each metric should be thought of as a diagnostic tracer. The more tracers you combine, the more precise your diagnosis will be.

Next time

There is lots of other interesting data in the agent performance metrics which you can use to further analyze build performance. In my next post, we’ll look at the Usage records section.