Deployment automaton, aviation style.August 12, 2012 — Dan Gordon
Forester analyst Glenn O’Donnell uses a good analogy when he speaks to describe the importance and value of testing and refining your application deployments before pushing an application to production. To make his point he asks, “What if Boeing didn’t test it’s plane designs before you flew on them?”
So, as I sit here in a Boeing designed metal tube, hurtling through the air at 30,000 feet and 500 mph, coming back from talking to 3 customers trying to get their deployments under control, I can’t help thinking about this analogy.
Granted, untested planes that fall out of the sky can have a far more grave impact than a failed application deployment. They are not of the same level of consequence. But, from the more extreme risk scenario, we can learn how to better handle in the less extreme.
The way Boeing knows that they won’t mass kill their customers with their products is by rigorous design, testing, and refinement. They design for manufacturability (capture all the details of how to implement their product), implement it, test it, refine where problems are found, then test again. Lather, rinse, repeat. Along the way they design for failure, and test those designs as well.
On the application deployment side, the same rigor can help us to achieve production deployment results that work reliably, and are resilient to what could become failures.
Here are the parallels:
Design for manufacturability – Boeing knows that a success with a hand crafted plane that is unique in it’s construction does not help them to get commercial planes safely into the air. They need to be able to test consistent models in many scenarios, over time, with minor adjustments, knowing that the majority of the aircraft which has already passed testing does not change. Repeatability is essential. The same holds true for applications being deployed to varying environments, over time. The goal should be to maintain repeatability of the design and implementation so that the test results of earlier phases are relevant information for later phases.
Test early and test often – Boeing doesn’t wait until just the first commercial passenger flight to test it’s planes. Neither should we for deployment to production. Just like being able to construct a plane consistently allows Boeing to test the planes thoroughly, a consistent deployment model enables thorough testing of the deployment process, helping to evolve and harden it for eventual smooth, fail-safe production deployments.
Zero in on defects efficiently – If every time Boeing found a failure in a plane design they started troubleshooting from the first component of the plane, “Ok, take it all apart and let’s start over from bolt #1″. . . they would never bring new planes to market. Instead, Boeing uses many custom tools designed to allow them to efficiently isolate problems and resolve them, without entire redesigns and rebuilds. Similarly, in application deployment we should have tools that make troubleshooting complex deployments much more efficient, so we can also resolve issues and get to market as quickly as possible.
Design anticipating failure – Despite our best efforts, failure happens. Better to be prepared than surprised. Under seat flotation devices, emergency exists, oxygen masks. . . In application deployment, we should be able to define what is an acceptable failure, have a well defined and tested recovery or rollback plan, and be able to set rules about how to handle failure should it happen.
In summary, the rigor which plane design and manufacturing require are well developed to help ensure safe products (new airplanes) can be brought to market in a consistent manner. The forming discipline of application delivery automation can learn from the aviation industry how to evolve to fail-safe production deployments.