These interviews were being done with large and small companies in Silicon Valley at that time and as you can imagine, the amount of change that’s happened in all of these companies are enormous. Some are no longer around due to acquisitions and other circumstances, but below are a couple of nostalgic anecdotes I thought was worth sharing with you:
• One truly innovative multidisciplinary company, today regarded as the global role-model for all Internet companies with some 33,000+ employees, had at the time of the interview 280 employees – where 100 of them were software engineers and apparently only 3 worked in QA!
• A major disruptive company in the virtualization business later acquired by EMC had at this time 60 engineers – current number that I’m able to research is some 9000 employees overall.

The Agile Manifesto had just been published in early 2001 and in these interview-notes, I cannot find any reference to common commodity terms and terminology such as Agile methodologies, Scrum, Kanban, Continuous Integration and Continuous Delivery. Yet when reading through the notes, I was surprised to find that many of the problems, concerns and issues that I hear about today with respect to Agile enablement blockers working with Electric Cloud prospects on a daily basis in my role as the ElectricAccelerator Product Manager very much were problems, concerns and issues already 10 years ago! Below are some quotes I picked up, pretty much summarizing the entire material of notes:
• “We’ve spent millions of dollars to make builds go faster, yet we are not satisfied”
• “We’re struggling to get releases out in time, often due to slow and unreliable builds”
• “My software builds are taking longer and longer, due to the proliferation of platforms I need to support and the staggering growth in complexity.”
• “If you could speed up my build by a factor of 10x, that would be huge!”

To quickly put this in perspective compared to the software development world of 2012, today I was in a partner discussion where we discussed a prospect in the automotive industry that had 8-9 hour long nightly builds and 20m+ developer incremental builds – clearly impacting engineering productivity and prohibiting other Agile development methodologies to efficiently be put in place.

Based on these interviews, our two founders compiled a graph illustrating the key prioritized features and value propositions that Electric Cloud should be founded to implement and realize, and I think this is pretty interesting to share with you:

So with Electric Cloud’s flagship product ElectricAccelerator that were designed and built to implement the above top features – where are we today, 10 years later?
• 10x faster builds, requested as a desired feature in almost 90% of the interviews: For the vast majority of our customers, a 10x improvement of their full build times is today commonplace. In fact, in quite a few of our customer deployments we know of acceleration factors exceeding 20, or even 30 and 40x. Last week I actually heard of the most extreme acceleration factor ever with ElectricAccelerator – a 72x speedup bringing a 36h+ build down to less than 30m!
• More platforms supported: To make sure our product is as compatible as possible for the marketplace, the ElectricAccelerator product team are constantly reviewing which platforms are mostly being used in the software industry and considering additional platforms to support. At the time of this writing, ElectricAccelerator runs on all supported Windows platforms, a number of Linux distributions including RedHat, SUSE and Ubuntu, and also on Solaris for both SPARC and x86. All in all, there are 14 major supported platforms that our automated build and QA lanes are covering daily!
• Better Dependency Tracking: One of the major key differentiating technologies of ElectricAccelerator from any other software build solution out there is its ability to detect, track, visualize and react to complete and guaranteed dependency information, without any timing overhead. This is also what enables the massive parallelization and distribution of builds that in turn realizes the dramatic speed improvements.
• Scheduling and Self-service builds: This is deliberately not a feature that ElectricAccelerator was originally designed for to implement, but given all the demand that the marketplace raised for the capability to automatically manage the end-to-end build-test-deploy workflow once the software build speed problem was pretty much solved, Electric Cloud decided to implement and introduce ElectricCommander.

Given this full coverage in terms of the top prioritized features that ElectricAccelerator was originally set out to implement, what are the next area of problems to address for the product? Based on all the user data and knowledge we’ve gathered over the years about software build environments in particular but also software development best practices in general, this was a pretty easy question to answer for me as the Product Manager of ElectricAccelerator: Optimize productivity and efficiency for each individual software developer in the context of a scaled out Agile development methodology, by revolutionizing the way software developer incremental builds are being done!

If you find this above statement interesting and would like to know more details, please don’t hesitate to contact me at drosen (a) electric-cloud.com. In the meantime, stay tuned!

Imagine a situation that we refer to as Jenkins sprawl. Hudson/Jenkins sprawl is when there are multiple teams in an organization are all maintaining their own instances of Hudson/Jenkins which perform independently to perform tasks. Much like the well known VM sprawl, managing Jenkins sprawl can become a management nightmare.

There is a point when an organization outgrows Jenkins and needs to provide end-to-end automation and data collection. The answer doesn’t need to be to persuade every individual team using Hudson/Jenkins in their lab or on a desktop to migrate to another solution. There is another, less painful way.

ElectricCommander makes it easy to embrace the existing Hudson/Jenkins instances, allowing them to do what they do well–perform team level continuous integration (CI). ElectricCommander can take care of the rest of the tasks needed for end-to-end enterprise scale automation.

The ElectricCommander command line interface is the key to this. When a build in Hudson/Jenkins starts, invoke ectool from the build script to tell ElectricCommander a build has started and to initiate a workflow process in Commander. When the build finishes, use ectool to tell ElectricCommander to move the process workflow to the next stage. This allows teams to continue using small scale CI systems, while taking full advantage of the enterprise class features of ElectricCommander.

The approach doesn’t just apply to Hudson/Jenkins, nor just for CI processes. Having the ability to control ElectricCommander easily from external tools is a feature I find truly compelling, especially as the command line interface and perl API are fully featured, providing much more than a limited subset of calls necessary to automate your system.

Driven by a confluence of business and technological trends, the move to DevOps will continue to gain traction. To keep pace with this momentum, it’s incumbent upon the development and operations teams to work together more closely. This requires cultural, process, automation, and other changes to existing behavior.

When evaluating automation solutions and other technology, it’s wise to adopt a system that fosters integrated Dev and Ops collaboration. Each team has much to offer: Dev possesses essential experience and knowledge that can lay the foundation for a successful DevOps architecture, and Ops is critical to deploying and running the application.

To help evaluate a DevOps solution, consider these 10 questions:

1. Can it easily model the development and delivery workflow, including the entire build-deploy-test-release process?
2. Can it leverage existing scripts and add value around them?
3. Is it flexible, with support for any language and methodology, such as Agile, waterfall, and others?
4. Does it provide out-of-the-box integration to many of the common tools in the organization’s tool inventory, such as build, deploy, test, release, and so on?
5. Can it integrate to any infrastructure platform, including physical, virtual or cloud?
6. Can multiple tasks be run in parallel? Is it easy to model parallel jobs?
7. Can manual transition approval steps be configured?
8. Can it enforce role-based access control between teams?
9. Is the solution able to scale to the entire organization, with hundreds of users, resources and concurrent tasks/jobs?
10. Does it provide real-time and comprehensive visibility into the status of the workflow process and jobs?

For all of the reasons we’ve been describing so far, it’s a good idea to start treating Dev and Ops as equally important participants in a single process. This means embracing the Build-to-Delivery cycle as a single, all-encompassing workflow, rather than continuing to treat it as many disparate procedures. It also requires aligning the previously incongruent organizational goals of both the Dev and Ops teams. This realization is the first step towards making the vision of “Continuous Delivery” and “Continuous Deployment” a reality.

As we stated earlier, the most successful DevOps implementations proceed from a Dev-first perspective. There are three primary justifications for this approach.

1. Take advantage of Dev’s comprehensive understanding of the application model

As the creator of the application, Dev teams are intimately familiar with what will be deployed. Dev can supply Ops with unparalleled insight into critical internal software attributes, such as:

• Application structure
• Necessary ancillary software and services
• Configuration settings
• Optimal application-specific runtime patterns

Understanding these characteristics is a prerequisite for a smooth software deployment process.  By using a system that can leverage the Dev application “smarts”, Ops can learn the application model and its infrastructure dependencies quickly and accurately.

2. Benefit from Dev’s experience with the application deployment process

As part of their normal responsibilities, Dev teams conduct frequent application deployments into many development environments. These tasks repeatedly occur at many points in the Dev cycle, such as:

• Software development
• Integration testing
• Functional/system testing
• Acceptance testing

In response to these demands, Dev usually has a great deal of experience with deployment. They have to, because they wouldn’t be able to fulfill their responsibilities without this knowledge. This expertise is very helpful when it comes time to place the solution into production. Examples of Dev’s operational proficiencies include:

• Installing and adjusting platforms such as operating systems and middleware to comply with application requirements
• Properly configuring the applications
• Fine tuning the applications to the specific functional need of the Dev tasks

Ops teams can leverage this expertise instead of starting from scratch.

3. Let Dev provide critical end-to-end visibility for minimizing potential deployment and production problems

Recall from earlier that a primary goal of the Ops team is to keep things running smoothly. However, with the frequent deployments mandated by Agile and other factors, this goal is at risk. The results can be downtime, poor performance, errors, and a host of other undesirable outcomes.

As the owner of the application software, Dev teams have complete comprehension of what changes have been applied, their history, and who approved these updates. Pairing this proficiency with Ops’ knowledge of the enterprise’s infrastructure yields complete end-to-end awareness.

Come back next week for the conclusion of this 5 week blog series.

Since the realities (mentioned in the two previous blog entries) have been in place for years, why is there such newfound momentum towards integrated DevOps procedures? The answer can be found in a series of business and technology-oriented fundamentals that have particular urgency today.

Business dynamics

Nearly every enterprise is facing extraordinary competitive pressures. This is a major cause behind the move to Agile practices. Yet at the same time, no business can afford to speed up their software delivery process only to regularly implement unreliable, sluggish, buggy applications.

In the past, software was delivered relatively infrequently. This gave operations teams ample time to prepare and meet the inherent performance and stability mandates for enterprise-grade applications. Today, Agile practices have completely disrupted the traditionally leisurely pace of deploying software. Ops teams are now placed in the unenviable position of serving as a bottleneck to business agility.

Technology dynamics

Developing software today means juggling more platforms, teams, tools, and infrastructure than at any point in history. Meanwhile, Agile practices often translate into hundreds of delivery iterations per month.

In reaction, many Dev teams have invested in powerful infrastructure to support the software release process. These solutions provide highly desirable benefits, including:

• Holistic and coherent enterprise-wide processes
• End-to-end visibility
• Capturing and disseminating best practices
• Close loop analysis and reporting
• Consistent security policies

For their part, Ops teams are leveraging the power of virtualization and cloud computing to seek out new efficiencies.  No longer is infrastructure holding up delivery of applications: instead, it’s now up to the teams themselves to work together to quickly get applications to production.

Finally, with the growing need for audit and compliance, it is critical that Dev and Ops have complete visibility into each other’s processes. Businesses now demand answers to challenging questions such as:

• What changes went into this application?
• Who approved these alterations?
• What packages comprise this application?

Without complete understanding of the entire Dev and Ops process, it’s not possible to conclusively answer these inquiries. Clearly, the time has come to bridge the gap between the Dev and Ops teams, which is what we describe next week.

Come back next week for the fourth installment of the 5 week series entitled Three Reasons to Concentrate on Dev in a Well-Planned DevOps Strategy.

And a very exciting debate broke out at this event:  Who drives DevOps – Dev or Ops? (sidebar note – only in Silicon Valley can you get  100+  passionate Ops/Dev people to show up to a DevOps meet-up)

The Ops folks – about 80% of the crowd – believed strongly that it was they who are driving DevOps to be a reality. These Ops folks believe that Ops can do better to enable Devs by putting the abilities of Ops at Devs immediate disposal, but not without them also getting some of the responsibility (Devs get to be on-call for their changes). So these Ops are driving DevOps so that they get out of the critical path of getting quality applications released to customers.

The Dev folks had a slightly different take. They see it as Ops holding back the customer delivery of the work they’ve done. Their work doesn’t get to users quickly enough and so they are frustrated, and have taken matters into their own hands.  Some on the Dev side look at DevOps more as NoOps (a VERY controversial term) indicating that with proper PaaS, there should be no need for traditional Ops.

As vendors in this space, we have seen the scars from the battle on both sides of this equation, and we believe that regardless of who is driving it, the DevOps movement has both sides working together towards a single goal, and this is a good thing for the business. DevOps is good for everyone – higher quality and differentiated applications makes it faster to the customer – and Dev and Ops can share in the success.

Who drives DevOps in your organization?

For decades, Development and Operations organizations have been divided. This gulf exists for one key reason: Development’s mission and focus is creating value by producing change in response to customer and market requirements., yet Operation’s mission and focus is creating value by keeping applications and services running reliably so that customers can depend on them. Unfortunately, with traditional software development and delivery methods, a higher change frequency often directly correlates to a higher risk to application reliability.

Cultural differences regarding change

While it’s dangerous to engage in broad stereotypes, it’s fair to say that Dev tends to be more risk-taking while Ops is more risk-averse. After all, the Dev team is paid to create change and the Ops team is paid to keep the enterprise running.

Through the development of Agile and lean software development techniques, Dev has discovered that smaller and more frequent releases are better for producing higher quality software which is more likely to meet customer expectations and needs. Dev has proven that higher frequency releases leads to meeting their goal of creating customer value.

Meanwhile, experiences in software delivery have made Ops teams all too aware that the majority of downtime happens as an unintended side effect of making changes to software. In response, Ops has developed many mechanisms to guard the gates to software changes, such as heavy change management and encouraging fewer releases.

The technology divide – process and tools

Driven by the desire for heightened business dexterity, many Dev organizations have either already made the transition to Agile practices, or are seriously considering this type of conversion. Agile’s primary goals include smaller releases, more frequent delivery of working software, and a dramatically shorter time to market. In order to attain these ambitions, Dev tools have been growing more sophisticated, with heavy emphasis on boosting development and testing speed as well as improving collaboration among teams. The evolution of Dev tools is enabling software to be produced and delivered at an ever increasing pace.

On the Ops side, Dev’s augmented delivery pace has led to an increased demand for system resources. Ops tools have also grown in their sophistication to meet these needs. By automating system configuration, virtualizing entire software stacks, and enabling resources on demand via the cloud, Ops has evolved their tools and processes to assist the delivery of resources faster and more consistently. Yet despite all the enhanced efficiencies supplied by modern Dev and Ops tools, it’s still difficult to deploy software in a consistent, repeatable, and reliable manner. To bridge this chasm, the Dev and Ops teams must collaborate much more than before. This newfound emphasis on coordination is driving the move towards DevOps.

Come back next week for the third installment of the 5 week series entitled Why is the Migration to DevOps Happening Now.

Introduction

DevOps has become an increasingly popular technique for shepherding software from the design phase through development and testing, and then all the way into production. The business value of DevOps is quite profound: DevOps reduces software delivery times, improves application quality, and enhances the productivity of the development and operations teams.  In recognition of this trend, many organizations are interested in automating their software release and deployment processes. However, by focusing exclusively on the “production” release/deployment process – a task primarily managed by Operations (Ops), organizations are not fully leveraging the value that Dev can bring to DevOps.

DevOps Overview

What constitutes DevOps? First, Dev (shorthand for ‘development’) signifies the teams and procedures involved in creating software. The exact meaning of Dev varies among organizations, but it typically incorporates:

• Software development
• Build, compile, integrate
• Quality assurance
• Software release
• Support in deployment and production tasks as needed

Secondly, Ops (shorthand for ‘operations’) refers to a wide swath of critical responsibilities such as:

• System administration
• Provisioning and configuration
• Health and performance monitoring of servers, key software infrastructure, and applications
• Change and release management
• Other infrastructure support

Thus, merging Dev and Ops results in a blend of principles, guidelines, and best practices that deeply involve both of these groups. Furthermore, DevOps is bolstered by technologies such as release management and deployment tools.

Come back next week for the second installment of the 5 week series entitled Why is there a divide between Dev and Ops.

This is where pushing a system to its extreme–and then a little more–may do the trick.  The extreme goal approach is a bit like pulling the band-aid to expose all the wounds, but it also directs the team to the  fixes that can provide the most  dramatic results in the shortest period of time.

Recently, I got to hear about how one of our customers has perfected this practice in their organization.

The GM of the software team, upon taking on the software mantle realized that the development methodologies did not let the company deliver software fast enough to their customers. Everyone knew that Agile was the answer, but how do you get there and what goals do you set?

GM:  “What is the fastest software delivery cycle that we can support today.”

Dev team:  “We do it every 6 months.”

GM:  “What if we go Agile?”

Dev team:  “Well, we may be able to get it down to 2 months”

GM:  “What about every 15 days”

Dev team:  “Impossible, Everything would break”

“Well–isn’t that what we need to do fix our process?”

And that is exactly what happened at this organization. As the GM describes it– many things did break in 10-day cycle. The team quickly identified all the key bottlenecks–the build system, test systems, limited team collaboration, unoptimized infrastructure setup, etc. They then employed extreme goal orientation to enforce strict prioritization of effort. As the GM put it “Now, if you had a opportunity to automate task A that saved 10 minutes or accelerate task B that saved 2 hours–it was clear where the team put it efforts.”

It was this extreme focus on the 15-day delivery goal that made this team go from waterfall to a fast Agile shop in less than 1 year.  Not too shabby for a several hundred person team embarking on Agile!

There is a moral to this story. You can set incremental goals, make incremental changes and ultimately reach those goals–in due time.  But if you want accelerated progress, here’s another approach–set an impossible goal, get out of the way and let the smart people make it happen.

p.s.–JFK’s Moonshot speech is a similar example of an  extreme goal setting. For more context, watch this video http://www.youtube.com/watch?v=ouRbkBAOGEw

We couldn’t have said it any better.

“Continuous delivery will reach farther into development process automation and the number of companies who can keep up with the new demands will continue to shrink. As a result, it makes sense to take time to choose your CI platform with particular care. CI builders can be difficult to swap out if you made the wrong choice.”

As a vendor that automates mission-critical software development and delivery processes at many enterprises, we realize it is precisely the choices that our customers have made while automating their CI process, that are now enabling them to extend their solutions to Continuous Testing, Deployment and even DevOps.

To learn more about how Electric Cloud supports DevOps, read the Ovum report here – www.electric-cloud.com.