The customer (for privacy reasons, we shall not name them here), creates cutting edge software used by millions of web users/consumers around the globe.  Their software is written by a large development team–once coded, the software is built, tested and then promoted/deployed to their production data centers. And this process, even for this very sharp engineering team, is quite a tedious one.

Building with Maven, running system, performance, and acceptance tests, and packaging for deployment is complex enough; working with these tools, different teams, disparate products and varying skillsets only added to the complexity. The process would take them a week or even months (3 months is what the customer actually calculated on average) to get from development to production.

But their business demands and goals are very clear. There is only way to win in the marketplace—iterate and deliver fast.  If you need to succeed in today’s fast moving market, you need to be more aligned and adaptable to your customer needs.

And that’s where Continuous Deployment comes in. The manifesto for Agile development states: “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale.” This is at the core of Agile, because it is only in the act of delivery that we close the loop with the users and initiate feedback. And our customer has taken this notion to heart.

Today, the customer has automated their entire process using ElectricCommander. Every one of their development, testing and packaging tool is completely integrated and automated by the ElectricCommander solution. The customer has built workflows that sequence and manages the entire process. As software is written, compiled, tested and deployed to the various environments along the process chain. Tedious manual operations have been eliminated, and errors are detected earlier in the cycle. All this gives the customer good visibility into their software delivery status – what application is in which state of their delivery pipeline.

Best of all, the entire process – from development to Ops can be executed in less than 10 minutes! Now the customer is able to experiment with their software in a way most software organizations can only dream of. And that makes them truly competitive in the marketplace.

Just another happy day in the life of an Electric Cloud customer.

Non-existence Conflicts

One obvious enhancement is to ignore conflicts when the two versions are technically different, but effectively the same. The simplest example is when there are two versions of a file which both indicate non-existence, such as the initial version and the version created by job C in this chain for file foo:

Suppose that job D, which falls between C and E in serial order, runs before any other jobs finish. At runtime, D sees the initial version, but strictly speaking, if it had run in serial order it would have seen the version created by job C. But the two versions are functionally identical — both indicate that the file does not exist. From the perspective of the commands run in job D, there is no detectable difference in behavior regardless of which of these two versions was used. Therefore emake can safely ignore this conflict.

To read the rest of this post and find out the other special cases to improve performance, check out Eric Melski’s blog here.

What is the Batch API?

In a typical ec-perl script, you issue requests to the server one at a time and process each result before issuing the next request. For example:

my $xpath = $ec->getProperty("foo");
my $fooVal = $xpath->findvalue("//value")->value();
... some logic with $fooVal ...
$xpath = $ec->getProperty("bar");
my $barVal = $xpath->findvalue("//value")->value();
... some logic with $barVal ...

Each of these requests is a round-trip to the server. Thus, the script blocks twice waiting for responses. The Batch API allows you to aggregate multiple requests, send them all together in a batch, and wait for the aggregated response. The example above could be implemented like this:

my $batch = $ec->newBatch("parallel");
my $fooReqId = $batch->getProperty("foo");
my $barReqId = $batch->getProperty("bar");
$batch->submit();

# Get the value of the foo property from its response. We can't be
# lazy and do the same "//value" xpath query as before because
# that'll do the search from the root of the entire xml response,
# which includes *both* responses. Instead, we do a relative
# query starting at the appropriate "response" node.
my $fooVal = $batch->findvalue($fooReqId, "property/value")
                   ->value();
... some logic with $fooVal ...
my $barVal = $batch->findvalue($barReqId, "property/value")
                   ->value();
... some logic with $barVal ...

The Batch API addresses three shortcomings of the traditional API:

1. It allows you to “batch up” multiple requests, reducing the number of round-trips to the server.
2. It allows you to run multiple requests in parallel, as shown above in the getProperty example.
3. It allows you to run multiple requests in a single transaction; all succeed, or no change is persisted in the server.

Each of these shortcomings are handled by a batch “mode” specified when creating a batch object:

serial

run the requests in the batch one after the other, serially, in different transactions. If a request in the batch fails, the other requests will still run. Each individual response indicates success or failure for that request.

parallel

run the requests in the batch in parallel, in different transactions. If a request in the batch fails, the other requests will still run. Each individual response indicates success or failure for that request.

single

run the requests in the batch one after the other, serially, in the same transaction. If a request in the batch fails, the transaction is rolled back, thus reverting all requests in the batch. Walk through each response to find the one that failed and why; all subsequent responses have a generic “batch failed” error to indicate that those requests didn’t get run at all because of the earlier error.

Performance Comparison

The following tests show how much performance gain the batch api provides for various scenarios.

Procedure Creation Test

In this test, we create procedures of various sizes with the batch api (single transaction mode) and without the batch api. All ec-perl scripts establish an ssl connection with the server when submitting the first message, regardless of whether that first message is an ordinary request or a batch of requests. The test results reported below factor out this cost by issuing an innocuous getServerInfo call, then measuring the time it takes to issue the createStep requests.

This graph shows that using the batch api reduces procedure creation time by about 50% for medium to large procedures and the time difference is negligible (less than 1 second) for procedures with 10 steps. In this scenario, the local agent on the Commander server issued the create step requests in a job. It’s interesting to note what happens if the machine issuing the requests is on a WAN; to emulate that, I ran the test from a home network VPNed into Electric Cloud with a Commander server in our office:

As one would expect, the round trip costs are somewhat larger now, so using the batch api reduces procedure creation time by about 66%. Again, not a lot of impact for small procedures (in this case, it took about 1 second to create such a procedure either way). Also, because the batch api uses one round trip to create the procedure, the WAN versus local-agent numbers are virtually the same, showing that most of the time is spent in server processing and the I/O costs are negligible in comparison:

Parallel GetProperty Test

In this test, we time how long it takes to get varying numbers of properties with a parallel batch and without the batch api. As for the procedure creation test, we factor out the cost of establishing the ssl connection to the server.

The difference here isn’t as dramatic as in the procedure creation case, but it’s still quite significant for medium and large numbers of getProperty calls with roughly a 30% time savings when getting properties on the local agent and about 55% savings when getting properties over a WAN.

Conclusions

The batch api provides significant performance benefit in both LAN and WAN environments for medium and large sets of requests. For small sets, the benefit is not very high, but it’s certainly no worse than issuing requests one at a time. Also, for scripts that contact the server repeatedly with a few requests every time, the little savings from each small batch request can add up. Of course, if it’s possible to structure such scripts to have a few large batches of requests instead, you’ll achieve the best speed-up.

For more details and examples, see the Using ectool and the Commander API section of online help.

But then, inevitably, invariably, you run into parallel build problems: incomplete dependencies in your makefiles, tools that don’t adequately uniquify their temp file names, and any of a host of other things that introduce race conditions into your parallel build. Sometimes everything works great, and you get a nice, fast, correct build. Other times, your build blows up in spectacular fashion. And then there are the builds that appear to succeed, but in fact generate bogus outputs, because some command ran too early and used files generated in a previous instead of the current build.

To read the rest of this article, check out Eric Melski’s blog here.

It’s a sad fact that in most enterprises, the process of building, testing, and deploying software is a‘black box’: very few people are able to explain exactly what occurs when software is delivered. Thisis a natural outgrowth of the ‘one-off’ manual processes that are so prevalent. There’s limited captureof inputs, and even less analysis of the results of this process. Any automated logic that does exist islocked away in homegrown scripts.

Automation encourages and facilitates migration towards a meta-driven software delivery process. Allresults are captured and summarized at the appropriate granularity, whether at the executive, team,or job level. Keeping track of this critical information makes it much easier to identify and remedybottlenecks, errors, and other complications that can sabotage any Agile initiative.

Any feedback is of course welcome (I sure hope my team won’t correct me too much).

Read the other parts of the series here:

1. Introduction
2. Empowerment
3. Goals
4. Feedback
5. Respect

One thing I try to do at every opportunity is to give immediate feedback for a job well done. “Great idea”, “Awesome debugging foo”, etc. When you see an interesting tweet post or email, take the minute to let that person know you appreciated that post. Try it, not just within your organization, but with anyone you interact with. Give the praise publicly, so people can share when good things happen. Without this, I found that people end up eventually feeling “Well… nobody cares about me doing this, so I might as well stop”… and then they do. So sad, so avoidable.

I am also trying to give negative feedback as early as possible. Now, I know it’s never fun or comfortable and, thankfully, I haven’t really had to do that a lot in my career. I think we all agree that feedback should be constructive, working towards fixing the problem. There may be situations where that is not possible, at which point the best that can be done is to find a mutually beneficial resolution (uncomfortable situation I am sure).

So how does it work? Excellent, everybody holds hands and sings kumbaya for most of the day (Well… don’t tell my boss). But, seriously, my team continues to amaze me with great results, unique ideas and a willingness to go the extra mile or two.

Read the other parts of the series here:

1. Introduction
2. Empowerment
3. Goals
4. Feedback
5. Respect

That level of individual decision-making freedom implies that a wrong decision isn’t grounds for immediate disciplinary action. We all are human, and make mistakes (Just ask my kids…They’ll tell you that I don’t do anything right). So let’s fix the mistake as quickly as possible and learn from them. What is not acceptable here is making the same mistake over and over again.

It is also important to be available for consultation, and that is a hard one. Time flies, and there are so many questions to answer. The only thing I’ve found that works: be proactive and constantly vigilant. I made a commitment to empty my email inbox by the end of each day.

Read the other parts of the series here:

1. Introduction
2. Empowerment
3. Goals
4. Feedback
5. Respect

Get engaged with the various goals at every level. I am lucky enough to work at a company where I can talk to the CEO any time to pitch an idea or get feedback. And so can anyone else in the company. No guarantee that it will be accepted/implemented, but it’s a good start (that’s what I like about small companies).

Goals change on all levels all the time, and it is hard to balance one layer of goals against the other. For the product, fixing this list of nasty issues is the best, yet the engineer that would be best suited for that has been in maintenance mode forever and really needs a break. What to do, what to do… Well, I found that personal interaction works pretty well, again with the result of a agreed upon goal. Something along the line of “I know you don’t want to fix another Windows device driver bug, so we will make sure that you get time to train your colleague on this, so next time you are not in the hot seat.”

So how to communicate goals efficiently? Well, I am still struggling with that one at times (I know I can do better). There is of course the traditional team meeting, there are five minute standup meetings, there is e-mail, the wiki, personal conversation (I hope I haven’t missed an obvious one). Different methods work better with different people, so I am trying to do all of them as appropriate. Kind of a squishy answer, which really offends my sense for black and white answers, but that’s all I have.

So what goals do we have at Electric Cloud and my team?

• We want to make our customers successful.
• We want to have fun doing it, or at least see the huge value in happy customers when it is not a lot of fun.
• We want to give team members a chance to explore new areas (sometimes voluntarily, sometimes not… just kidding). We do have a policy that engineers can have a day each month to explore topics/interests at their own leisure that are not directly job related. This gets their minds thinking in new and creative directions, and helps them avoid getting bogged down in old and frustrating issues.
• And let’s not for get my early retirement… I may have mentioned that.

Read the other parts of the series here:

1. Introduction
2. Empowerment
3. Goals
4. Feedback
5. Respect

To me, all of it starts with the company culture. At Electric Cloud, we have a few guiding principles that help us make the right decisions:

• It’s ok to make mistakes. We want to take risks, and, if something turns out badly, let’s fix it. No blaming, no finger pointing.
• We want to know about problems. To be able to fix mistakes, we need to recognize them. So everybody is encouraged to speak up when they see a problem.
• Execution + Consultation = Success. Make decisions you can make comfortably, and consult with other team members when you need help. Consult first with whoever is most likely to have a problem with your decision.
• Get plenty of input, but move forward quickly. Big decisions warrant a greater level of input, but it’s important that you don’t get stuck in the input gathering/consensus building stage indefinitely. Remember… It’s ok to make mistakes (as long as you don’t run the company into the ground).
• Operate under the “Reasonable Person Principle.” If you don’t understand why someone else is acting/responding in a particular way, remember that that person is likely to have a good reason for whatever they are doing. Instead of assuming your seemingly-crazy colleague is just plain wrong, pause and try to understand their reasoning (This works up to a point… there may be one or two unreasonable people out there).

Measuring Success: What Makes Us Think This Works?

Of course, there are lots of theories… and many fewer successes. As the old adage goes “The proof of the pudding is in the eating”.  I’m glad to report that, so far, the recipe above has proved good eating for Electric Cloud.

• Our customers are really happy. A recent customer survey found that 87 percent of our customers are very satisfied or satisfied. We are working hard on getting those mostly satisfied up to at least satisfied.
• Our team turnover is very low. My team right now has an average tenure of 4.5 years. Over the 10 years of Electric Clouds existence, my team has lost four engineers from leaving. And yes, I did lose other engineers to other groups in the company, and I didn’t count retirement.
• Our product keeps getting better. We are able to produce four on-time feature releases per year. We have produced 12 feature releases at nearly three-month intervals for the past three years (… let me check my math… yes, that’s right). We also produced 15 patch releases in the same time frame.
• We are stomping out our bugs. Our bug open-to-close stats show that in the past year we have closed four bugs for every three bugs created. While our source base, as well as our customer base, keeps expanding, our overall open bug count has remained steady/declined.
• People know we are here to help. Within the organization (from marketing and sales to product management and support), the engineering team is well respected, and frequently called upon to provide help, guidance, support and even hugs (when requested).
• We are growing even in hard times. Our customer base keeps growing, and our existing customers consistently renew support contracts and expand deployments.

So yes…It seems to work. But how does this manifest in our daily life here at Electric Cloud? To me, the day-to-day business is driven by several factors. Read on to find out more:

1. Introduction
2. Empowerment
3. Goals
4. Feedback
5. Respect