What Is DevOps?

The goal of DevOps is to build better, faster and more responsive software by bringing Development and Operations teams together. DevOps is not a methodology or a suite of tools but a cultural shift to remove the barriers between Dev and Ops in order to meet the need for shorter and more frequent software deliveries. This will allow your organization to respond in a more agile manner to changing business requirements. The DevOps cultural shift depends on continuously optimizing workflow, architecture, and infrastructure in order to deliver high-quality applications.

From the Agile Model to the DevOps Model

Software developed with the Agile Model adheres to the following four basic priorities stated in The Agile Manifesto:

• Individuals and interactions over processes and tools;
• Working software over comprehensive documentation;
• Customer collaboration over contract negotiation; and
• Responding to change over following a plan.

Agile development takes a test-first approach, rather than the test-at-the-end approach of traditional development. In agile testing, code is developed and tested in small increments of functionality. Agile project management promotes frequent interaction between an IT department and business users and tries to regularly build and deliver software that meets business users’ changing requirements. The flexible approach of the Agile Model helps to ensure a streamlined software development process that responds quickly to customer demand in a wide variety of use cases.

Agile software development can be implemented with a number of methodologies, including Scrum, Kanban, Scrumban (a mix of Scrum and Kanban), extreme programming, and many more. In the Scrum methodology, the development effort is broken down into a series of short ‘sprints,’ or iterations, typically lasting 2 to 4 weeks. The goal of each sprint is to create potentially shippable software by adding new or enhancing existing application features.

The DevOps Model is the ‘Agile Model on steroids,’ which extends the cross-functional Agile team made up of software designers, testers and developers to include support people from Operations. The focus of this expanded agile team is the overall service or software your company delivers to its customers, instead of just “working software” (the 2nd of the 4 Agile principles listed above).

Building a successful continuous, two-way DevOps software pipeline between your company and your customers means creating a DevOps culture of collaboration among the various teams involved in software delivery (developers, operations, quality assurance, business analysts, management, and so forth), as well as reducing the cost, time, and risk of delivering software changes by allowing for more incremental updates to applications in production. In practice, this means teams produce software in short cycles, ensuring that the software can be reliably released at any time. Where regular agile teams strive to deliver shippable software in 2 to 4 weeks, the ideal DevOps goal is to deliver code to production daily or every few hours.

Finding the Right DevOps Testing Strategy

The aim of most DevOps projects is to automate as many manual processes in their software delivery process as possible. Some roadblocks in a DevOps pipeline that lead to slow deployment include error-prone manual processes such as handoffs from a development group to a QA group, including ones that require signatures or bureaucratic approval. These kinds of handoffs mean there is a lack of shared ownership of the end product, which is contrary to the basic agile testing and development methodology that says all members of a cross-functional agile team are equally responsible for the quality of the product or the success of the project. Because of this, testing on an agile DevOps project is done by the whole team, including developers, testers and operations people. In the DevOps model, testing is done continuously—early, often and even after applications are in production.

Continuous delivery vs. Continuous deployment vs. Continuous integration vs. Continuous Testing

Pros and Cons of DevOps Continuous Testing

Continuous Testing needs to be a key element your DevOps testing strategy if you want to successfully implement a DevOps pipeline. Continuous Testing, which is often called shift-left testing, is an approach to software and system testing in which testing is performed earlier in the software lifecycle, with the goal of increasing quality, shortening long test cycles and reducing the possibility of software defects making their way into production code.

Continuous Testing is one of several continuous activities that should take place simultaneously in a DevOps pipeline, including

• Continuous Development;
• Continuous Integration (CI);
• Continuous Delivery (CD) and  
• Continuous Monitoring

Continuous Development or build automation is the first stage in implementing a Continuous Delivery DevOps pipeline. If your developers are practicing test-driven development (TDD), they’ll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology, TDD helps developers think through the desired behavior of each unit of software they’re building, including inputs, outputs, and error conditions. New features implemented by developers are then checked into a central code base prior to the software build, which compiles the source code into binary code.

Continuous Integration is a practice where members of a software development team use a version control system and integrate their work frequently to the same location, such as a master branch. Each change is built and verified by means of tests and other verifications in order to detect any integration errors as quickly as possible. With build automation, the software build happens automatically, using tools such as Makefiles or Ant, rather than when a developer manually invokes the compiler.

In the last stage of a Continuous Delivery pipeline, once an application passes all the required tests, it’s then released into production. For all intents and purposes, this means releasing every good build to users. The upside of Continuous Delivery is that it allows delivery of new functionality to users within minutes whenever it’s needed, as well as instant feedback to the DevOps testing team that, in turn, allows them to respond rapidly to customer demand. A big CD downside is a risk of releasing buggy software into production faster, which will alienate end-users and cause your company to lose customers. To mitigate the risk involved with continuous deployment, many DevOps teams also test in production, otherwise known as shift-right testing, in order to obtain user feedback and incorporate it into successive builds. This last phase is often aided by Continuous Monitoring tools (more on this below).

Multiple teams work together as a DevOps team, so DevOps has an impact on not only QA but also developers, operations, business analysts, production support, etc. Testing is a critical element in promoting cohesiveness and collaboration between the various teams, automation of everything possible. Testing also allows for continuous improvement in the test process delivered via iterations.

Developing your devops testing strategy

A successful DevOps testing strategy is one aimed at building, testing and releasing software faster and more frequently. If you’re lucky enough to start out in a “greenfield” organization without an established coding culture, it’s a good idea to try to create and automate your software delivery pipeline upfront. If you’re successful out-of-the-gate in creating a Continuous Delivery DevOps pipeline, your business will be much more competitive since you’ll be able to get higher-quality software into the hands of your users and customers faster than your competitors, and you’ll be able to react to business demand and change much more rapidly.

If you are adding a DevOps pipeline to an existing organization, where you start depends on your current development and testing practices and the bottlenecks in your software delivery process. These bottlenecks can include slow, error-prone manual processes as well as poor-quality, big-bang rollouts that fail in production, leading to unhappy users.

There are several ways to get a handle on the current state of your deployment processes, including using workflow visualization tools like flowcharts and business process maps to breakdown and understand your current delivery processes. One of the simplest visual process management tools you can use to help make these kinds of decisions is a Kanban board.

Kanban boards, like the one pictured below, are typically just sticky notes on a whiteboard that are used to communicate project status, progress, and other issues.

The main advantage of using Kanban in your DevOps roadmap is that it encourages your teams to focus on improving workflow. It does this by highlighting bottlenecks in the pipeline that cause work to slow work down or otherwise impact the performance indicators your entire DevOps team has mutually agreed on–such as application time-to-market, systems uptime, code quality, customer satisfaction, and so on.

Kanban also helps your Dev and Ops people to work in a collaborative manner, which they may not be used to doing. Once they start talking about how to eliminate duplicate work and how to optimize your company’s release management processes, you’ll likely see an evolution over a period of time into a single team and single workflow that includes all of Dev and Ops activities–and a transformation to a DevOps culture.

It’s also likely that developers and testers on your DevOps teams will have some familiarity with Kanban if they’ve been using Scrum or other Agile methodologies, and they’ll be able to help your Operations people get up to speed with the concept of flow, which basically is what Kanban is all about.

[Atlassian offers the following tutorial that Zephyr for Jira tool users can use to better understand Kanban: “Learn Kanban with Jira Software Tutorial”: https://www.atlassian.com/agile/tutorials/how-to-do-kanban-with-jira-software (Step-by-step instructions on how to drive a Kanban project with Jira Software)]

Best practices for continuous testing

Automate as Many Tests as You Can

Continuous Testing means testing early and often. This is obviously a major challenge for your DevOps team if the applications in your pipeline are constantly being updated via Continuous Integration, where the code is integrated into a shared repository on a constant basis. A best practice is to use test automation to eliminate much of the risk that comes with CI and to get quick feedback on application quality. Pairing CI with automation enables teams to easily test every new code iteration and reduces the opportunity for team members to add errors. Use the Test Automation Pyramid below as a strategy guide to planning your DevOps testing strategy. As shown in the illustration, the base or largest section of the pyramid is made up of Unit Tests–which will be the case if developers in your organization are integrating code into a shared repository several times a day. This involves running unit tests, component tests (unit tests that touch the filesystem or database), and a variety of acceptance and integration tests on every check-in.

Figure 1: Test Automation Pyramid

Use Pair Testing on Tests You Don’t Automate

Knowledge transfer via pair testing is another best practice and one of the most efficient ways to train new members and maintain DevOps pipeline velocity, assuming your team has senior testers or developers who are willing to share their knowledge and time with new team members. Pair testing is similar to pair programming, which is an agile software development technique in which two programmers work together at one workstation. In pair testing, two team members work together at one keyboard to test the software application. One does the testing and the other analyzes or reviews the test results. This can be done between a tester and a developer or a business analyst or between two testers with both participants taking turns at driving the keyboard. Pair testing is useful in breaking down communication barriers between developers, QA testers and application/platform operators and creating interdependency between people in functional silos– thus building strong workplace alliances.

DevOps Tools

As mentioned earlier, software applications pass through five different stages when developed in a DevOps pipeline:

1. Continuous Development
2. Continuous Testing
3. Continuous Integration
4. Continuous Delivery
5. Continuous Monitoring

Because DevOps is a cultural shift that promotes collaboration between development, QA and operations, there is no one single product that can be considered the definitive DevOps tool. Often a collection of tools, from a variety of vendors, are used in one or more stages of the DevOps toolchain. The DevOps tools used in each stage will vary from organization to organization, based on tools already in place, the skill level of your developers and testers, and the types of applications you’re deploying.

Here is a short list of some the most common DevOps tools and an explanation of how they’re used in each stage:

Continuous development with jira

The de facto standard for the agile DevOps software development and testing, Jira Software has powerful collaborative functionality that visually highlights issues as they move through the project workflow, which makes the Jira platform easy to use for planning development, tracking daily work and reporting on project progress.

Continuous testing with zephyr for jira

Although Jira Software is designed for issue, project and workflow tracking on IT projects, many DevOps teams are also using it for test case management so that development and testing teams can work together in one system. Test issues can be created, executed, tracked and reported on just like any other Jira issue in Zephyr for Jira, which has the same look n’ feel of Jira.

Continuous integration with Jenkins

Jenkins is a CI/CD server that runs tests automatically every time a developer pushes new code into the source repository. Because CI detects bugs early on in development, bugs are typically smaller, less complex and easier to resolve. Originally created to be a build automation tool for Java applications, Jenkins has since evolved into a multi-faceted platform with over a thousand plug-ins for other software tools. Because of the rich ecosystem of plug-ins, Jenkins can be used to build, deploy and automate almost any software project–regardless of the computer language, database or version control system used.

Continuous delivery/deployment with puppet or chef

Automated configuration tools such as Puppet or Chef allow you to avoid the problem of “snowflake servers” in your delivery/deployment environment. “Snowflake servers” are long-running production servers that have been repeatedly reconfigured and modified over time with operating system upgrades, kernel patches, and/or system updates to fix security vulnerabilities. This leads to a server that is unique and distinct, like a snowflake, which requires manual configuration above and beyond that covered by automated deployment scripts.

Using definition files (called manifests in Puppet, or recipes in Chef) to describe the configuration of the elements of a server is a key best practice in Infrastructure as Code, which allows environments (machines, network devices, operating systems, middleware, etc.) to be configured and specified in a fully automatable format rather than by physical hardware configuration or the use of interactive configuration tools.

Continuous monitoring with splunk or elk

Both Spunk and Elk are log monitoring tools that allow you to do data analysis on transactions that take place in your IT applications after they’ve been deployed to ensure uniform performance, availability, security, and user experience. Both Splunk and ELK supply a scalable way to collect and index log files and provide a search interface for users to interact with the data in order to create visualizations such as reports, dashboards or alerts.

Splunk is a paid service that bills according to the volume of your transactions. The ELK Stack is a set of three open-source products—Elasticsearch, Logstash, and Kibana—all developed and maintained by Elastic.

Devops testing tools

Using some or all of the following DevOps testing tools provides numerous benefits to DevOps teams– including improving your teams’ code quality, accelerating the time-to-market of software in your DevOps pipeline and providing fast and continuous feedback that will increase collaboration among development, testing, and operations teams.

The first tests that should be done in a DevOps pipeline are unit tests that developers write as part of the test-driven development (TDD) process. If your developers are practicing TDD, they’ll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology, TDD helps developers think through the desired behavior of each unit of software they’re building, including inputs, outputs and error conditions.

TDD tools include JUnit, one of a family of open source unit testing frameworks that are collectively known as xUnit. Used by Java developers to write repeatable cases that increase programming speed and improve code quality, JUnit provides features such as fixtures, test suites, JUnit classes, and test runners. TestNG is an enhanced version of JUnit that covers a wider range of test categories: unit, functional, end-to-end, integration, etc.

In addition to doing unit and component testing, many agile DevOps teams also practice test-first approaches such as acceptance test-driven development (ATDD) and behavior-driven development (BDD) for tests higher up the Test Automation Pyramid (described above). This allows testing to be repeated in increments as software components are assembled upon each other.

Cucumber is a tool based on behavior-driven development that encourages a culture of collaboration between developers, QA and non-technical or business participants on DevOps and other software projects. BDD bridges the communication gap between business and IT by writing test cases in a natural language that non-programmers and domain experts can read. BDD features are usually defined in a GIVEN WHEN and THEN (GWT) format, which is a semi-structured way of writing test cases.

Devops test automation

Choosing the right DevOps test automation tool is vitally important for your organization since the right software testing tool in the right hands can foster team collaboration, drive down costs, shorten release cycles, and provide real-time visibility into the status and quality of applications in your DevOps pipeline.

Test automation works by running a large number of tests repeatedly to make sure an application doesn’t break whenever new changes are introduced—at the different Unit-, API- and GUI-levels of the Test Automation Pyramid. In addition to Continuous Integration tools such as Jenkins described above, here are some other useful tools for building, testing and deploying applications automatically when requirements change in order to speed up the release velocity of your pipeline apps.