Why DevOps is Doomed – Ops teams are lost!

    [published on dzone.com at http://server.dzone.com/articles/why-devops-doomed-ops-teams]

    The problem between dev and ops is primarily a terminology, communication and respect problem resulting in poor operational support.  The two organizations say common things backed by different definitions that are not in agreement. For example, would ops define an “application” in Puppet the same way dev would define an “application” in Jenkins? If not, how would you automate or even communicate between the two for automated application deployments? Dev and Ops really have no concept of each other’s world, yet they assume the other side understands their view, or they expect that the other side should understand their view.

    I love the concept of DevOps and I am very optimistic about the movement’s value. However, I’m also very concerned about traditional IT leadership’s capacity to focus on the right goals to make DevOps successful. Bridging development and operations is NOT about dev teams utilizing a continuous integration tool like Jenkins or Bamboo. And it’s NOT about ops teams standing up a configuration management tools like Puppet or Chef. Both may be needed for your automation efforts, but DevOps is about bringing dev and ops teams together so people and tools from both realms are communicating with common terminology, data sources and objectives. As always, communicating and working together for a common goal is the challenge!

    • Developers tend to think infrastructure is pretty straightforward. “I can stand up a server at Amazon in seconds. These clowns at work take forever with the simplest requests.”
    • Systems Administrators tend to expect developers to understand the infrastructure their applications run in. “The developer said it worked on his dev server, so obviously we screwed it up in production. The dumbass doesn’t understand firewalls or our company’s network.”

    On average, developers know application code architecture and think they know systems architecture, but they DO NOT. On average, systems and network administrators have good diversity and know a lot of different infrastructure disciplines, and think they know application code architecture, but they DO NOT.

    So why would DevOps be doomed for failure?

    Web applications, services architecture and cloud providers have destroyed any hope of success for the traditional IT leadership sold on yesterday’s operational support model. There has to be a fundamental change to recognize that systems and applications are no longer static, documented operational models; they are dynamic release-time dependency models. And there has to be a systematic way for dev teams to communicate application architectures so ops teams understand them.

    Have you ever been asked to document application dependencies? If so, could you? If so, how long was it valid? Documenting a traditional three-tiered application is pretty easy. Documenting an application in a service-oriented architecture is only valid until the next code release  –As each release may utilize a new service end-point, dependent on a new network segment, dependent on a new database, dependent on a new data center in a different region. Good luck on managing the relationships for your ops teams!

    Application designs no longer have a universal hierarchy; the diversity and rate of change can not be easily modeled in a traditional database schema. Enterprise IT tools used to manage the environment provide little help as they expect a static hierarchical application model. ITIL and service catalog implementations also tend to expect a static hierarchical application model. The three-tiered app is gone with the introduction of web application, service architectures and cloud providers. It’s game over if you can’t define your applications, model it, and use that same data to automate the build, deployment and operations life cycle.

    The bottom line

    App maps look like a circuit board.

    Operations teams are lost and have no idea what an application looks like, how to model it, or how to support it. Nor have traditional enterprise IT solutions provided the tools to help model the web app and cloud era. Today’s dependency maps look like circuit boards.  If you zoom in, you only see some components of your applications dependencies.  If you zoom out, you see the circuit board but can’t read or understand any details.

    Let’s say your web application renders a page. For that simple transaction, your application calls multiple web services, each with multiple endpoints, each with multiple database dependencies.  Some databases may be dependent on nightly ETL jobs to provide valid data for your functionality.  Maybe the UI is rendered by a separate UI platform or portal with its own application, service dependencies and meta data database.  Now, let’s say the relevant applications, services, and databases are developed by five different dev teams across three different states.

    An event: some functionality in your application fails intermittently.  How does your ops team troubleshoot the problem and resolve it?  Is the “application” just the part your dev team developed, or is the application the whole “circuit board” of dependencies?  Can your app be described effectively in a knowledgebase, KB article, or wiki site?  Can the “circuit board” be effectively described in a CMDB or support tools?  If so, who out of the five dev teams is accountable for maintaining changes to it?  Is your ops team relegated to calling in subject matter experts from each team for troubleshooting?  Is your ops team able to be effective without a clear understanding of the application?

    To be successful, we have to enable our ops teams to manage the dynamic changes and complexity of today’s applications. Manual communication processes will fail, so we need to redefine the minimum bar for “automation.” Systems Administrators creating a bunch a scripts and standing up Puppet or Chef is not automation. Developers using Jenkins or Bamboo for continuous integration builds is not automation.  Automation has to link the application, build, and configuration management together.

    • “Automation” needs to be an architecture platform, not an individual tool or effort.
    • Automation “platforms” must bridge the technical communication gap between development and operational lifecycle tools, thus enabling organizational DevOps efforts.

    The key is establishing common data models and service architectures that enables the automation and a common communication language at a very technical level. If you have been following Willie’s posts on skydingo.com, then it should be clear why we think a CMDB architecture using an unstructured NoSQL technology like Neo4j is so valuable.


    1. Ed Grigson | |

      Great post which describes our daily challenges brilliantly. I’m an infrastructure guy learning more about Chef and Puppet with a view that that’ll solve some of our deployment issues by at least tying together the code and infrastructure releases. As you righly describe I have limited knowledge of our development/release process or the code that’s handled by it. The dynamic nature of both sides is creating new challenges – the difficulty is both articulating them concisely (as this post does) and having a solution to present. It’s not going to be a quick fix.

    2. Brian Kozumplik | |

      Another disconnect between Dev and Ops arises from different goals/external pressures, not just differing information sets as the article looks into. Ops generally measures itself by uptime and stability, scaling and metrics and monitoring coverage, and dev judges itself by features launched, initiatives coded or redesigned, and bugs closed. Different focuses naturally lead to people marching in different directions at times.
      Having one good “embedded” ops guy literally sit with the devs, attend and facilitate their buildouts, and understand their issues goes a long way. Thats called “devops” in some orgs. It works remarkably better than old school seperated ops/dev groups.

      • Willie Wheeler | |

        Brian, this is an important point. Dev tends to create change and ops tends to resist change. The trick (and somebody mentioned this on LinkedIn in response to this post) is to align team motivations so that features delivered count as a win for both the dev and ops team, and downtime counts as a lose for both dev and ops. It shouldn’t be so hard to achieve that alignment–after all, you need solid infrastructure and ops to deliver value to customers, and downtime often arises from problems with the software itself.

    3. Andy | |

      Really interesting post Paul – I like the emphasis on cultural/mindset change rather than just a tool-based approach.

      And I agree that productive implementations of DevOps might be beyond the grasp of many large organisations, but I don’t see that as a failing on the DevOps side necessarily… as Deming says “survival is not mandatory” :-)

      On the automation side, you seem to be re-defining the term to mean “holistic end-to-end automation” which is a far wider, deeper usage of the term. I’d prefer to keep “automation” as the simple notion that it currently has but emphasise that it’s necessary but by no means sufficient to achieve much of the promise of DevOps/Continuous Delivery.

      • Paul Jenson | |

        Thanks for the reply Andy. Yeah, sometimes in trying to articulate a point in writing you need to be a little more dramatic than normal. DevOps will live strong and some large organizations will even be successful with it. I don’t think I really want to redefine the term automation. However, I do want to stress that simply automating the old process within the dev and ops silos does not solve any of the communication gaps. It adds value in speed, consistency and scalability which is obviously good!

        With the end-to-end automation, I want to point out the potential for much greater value. Let’s face it, getting dev and ops guys in a room to talk through their communication problems is useless effort. But getting a bunch of dev and ops geeks to focus on integration points for their automation –now that actually has a good shot at dialogue that will bridge some communication gaps.

        • kish | |

          Interesting, we’ve had better results when both dev and ops didn’t specialize only in their roles.

          Building applications and trying to manage them together has eliminated the silo mentality.

      • Willie Wheeler | |

        I agree with the idea that automation is necessary but insufficient. One of the points that Paul makes from time to time is that it’s important to pick the right automation targets–don’t automate a process that shouldn’t even exist in the first place, for example.

        One of the commenters on LinkedIn characterized devops in a way that really struck a chord with me: integrated process, tools and data. I think that is exactly right. Automation is better when it supports the right processes (smaller scale automation may not be particularly integrated, but it should be embedded in larger scale automation that *is* integrated), and it’s better when it’s driven by data from a single source of truth.

    4. Konstnatin Kondakov | |

      The configuration file needs to be clearly residing outside the code base and then be distributed by config management systems like: cfengine, puppet, chef

    Leave a Reply