DevOps Toolchain Roundup in 2022

Original

About DevOps & Tools

Here are a few things to remember about DevOps and its tools：

• Goals need to be continuously improved.
• DevOps is not a pay-for-play.
• Tool adoption occurs in stages.

I. Program Tools

Why are program tools essential to DevOps?

• Sharing of goals.
• Transparency.
• Empowerment.

1. GitLab

GitLab is a web-based tool for the DevOps lifecycle. It provides a Git repository manager with a wiki, issue tracking, and CI/CD pipeline features and is licensed under the open source license developed by GitLab.

2. Tasktop

Tasktop allows all these tools to be added to Agile, ALM, PPM and ITSM, enabling unprecedented visibility into the entire lifecycle.

3. CollabNet VersionOne

VersionOne supports Scrum, Kanban, XP, SAFe and hybrid development methodologies and makes it easier to plan, track and report across teams, programs, software portfolios and enterprises.

4. Pivotal Tracker

The agile project management tool is preferred for developers to collaborate in real-time around high-priority shared backlogs.

5. Trello

Trello is a web-based, Kanban-style checklist application, a subsidiary of Atlassian. It is used mainly by teams to plan their work sprint.

6. Azure Boards

We use agile tools, including Scrum, Kanban, and other agile methodology dashboards, to better track and plan software projects.

These are an excellent way to visualize work, share plans, track progress, and ensure progress toward goals.

7. ZenTao

ZenTao lets you create unlimited tasks, assign tasks to different people, and prioritize each task based on its importance level. You can also break down your tasks into milestones and then track how your progress is going based on these milestones. ZenTao also lets you schedule different projects and then see what your team's workload is like with its project schedule views. You can also create dependencies between different tasks to track whether everything has been completed on time.

II. Issue Tracking Tools

Why is the issue tracking critical to DevOps?

• User response.
• Reduction of knowledge loss.
• Closed loop of feedback.

1. Atlassian Jira

Jira is an issue tracking product developed by Atlassian that provides bug tracking and agile project management capabilities.

2. JetBrains YouTrack

YouTrack is a browser-based bug tracking, issue tracking system and project management software developed by JetBrains, Inc. It supports query-based issue search, including automatic completion, batch issue processing, custom issue attribute sets, and the creation of custom workflows.

3. Zendesk

Zendesk makes customer issue tracking more accessible, and it is used by many companies, including Uber and Airbnb, as their issue tracking software.

4. ZenTao

ZenTao is an open-source project management software. Its main features are product management and test/CI automation that helps corporations manage their ALM process in order to ensure efficiency throughout all departments within a business organization.

ZenTao lets you create unlimited tasks, assign tasks to different people, and prioritize each task based on its importance level. You can also break down your tasks into milestones and then track how your progress is going based on these milestones. ZenTao also lets you schedule different projects and then see what your team's workload is like with its project schedule views. You can also create dependencies between different tasks to track whether everything has been completed on time. Last but not least, ZenTao allows you to track how much time has been spent on different tasks. You can also track the time spent on individual resources working on a task, which is very helpful when tracking multiple resources on a single task.

III. Software Configuration Management Tools

Why is source control essential to DevOps?

• Asset control.
• Reduction of transmission losses.
• Promotion of teamwork.

1. Git

Git is a distributed version control system tracking source code changes during software development. It is designed to coordinate work among programmers, but it can also track changes in any set of files. It focuses on speed, data integrity, and support for distributed, non-linear workflows.

2. GitHub

GitHub provides Git distributed version control and source code management features, as well as some of its features.

3. GitLab

GitLab's version control can help your development team share, collaborate, and maximize productivity with world-class source code management features.

4. Bitbucket

Bitbucket is Atlassian's web-based repository hosting service for source code and development projects with Mercurial or Git version control systems.

5. Subversion

Apache Subversion is a software version control system licensed under the Apache Development Source. Software developers deploy Subversion to maintain current and historical versions of files, such as source code, Web pages, and documentation.

IV. Build Tools

Why are build tools essential to DevOps?

• Consistent packaging effects.
• Automated error detection.
• Early detection of quality issues.

1. Maven/Gradle

Maven is an automated build tool primarily for Java projects. Maven can also build and manage projects developed in C#, Ruby, Scala and other languages. Gradle is an open-source automated build system that introduces a groovy-based domain-specific language based on the concepts of Apache Ant and Apache Maven.

2. MSBuild

Microsoft Build Engine, widely known as MSBuild, is an open source, free managed code and native C++ code building toolset that is part of the .Net Framework, and Visual Studio includes MSBuild as a build tool.

3. Rake

Rake is a software task management and automated build tool. Users can specify tasks, describe dependencies, and specify and describe dependencies for groups of tasks in the same namespace.

4. JFrog Artifactory

JFrog Artifactory is a tool for saving the binaries generated by the build process for distribution and deployment. Artifactory supports multiple package formats such as Maven, Debian, npm, Helm, Ruby, Python, and Docker.

JFrog provides high availability, replication, disaster recovery, scalability, and integration with many proprietary and cloud storage products.

5. Sonatype Nexus

Sonatype Nexus is a repository manager. It can proxy, collect and manage dependencies so that you don't have to worry about piles of jar files. It makes the distribution of software much easier.

Internally, you can configure build tools, publish artefacts to Nexus, and then other developers can work with them.

6. NuGet

NuGet is the package manager for .Net. The NuGet client tools provide the ability to produce and consume packages.

The NuGet Gallery is the central repository used by all package authors and consumers.

V. Testing Tools

Why are testing tools critical to DevOps?

• Focus on quality.
• Improve product confidence.

1. JUnit

JUnit is a unit testing framework for Java. jUnit plays a very important role in test-driven development and is a member of the xUnit (derived from SUnit) family of unit testing frameworks.

2. xUnit.net

xUnit.net is an open-source unit testing tool for the .Net Framework, developed by the original authors of NUnit.

3. Selenium

Selenium is a framework for testing web applications. Selenium provides a replay tool for writing functional tests, eliminating the need to learn a scripting language.

4. Jasmine

Jasmine is a JavaScript testing framework in open source. It aims to run on any platform that supports JavaScript without interfering with the application or IDE and with an accessible syntax. It is influenced by other unit testing frameworks such as Screw.Unit, JSSpec, JSpec and RSpec.

5. Cucumber

Cucumber is a tool that supports behaviour-driven development. At the heart of the Cucumber BDD approach is a language parser called Gherkin that specifies the expected software behaviour in a logical language the user understands.

VI. Continuous Integration (CI) Tools

Why is continuous integration and deployment essential to DevOps?

• Rapid feedback.
• Reduced defects and waiting times.

1. Jenkins

Jenkins is a free automation server of open source that assists in automating the non-human part of the software development process. It is a server-based system that runs in a Servlet container, such as Apache Tomcat.

2. CircleCI

CircleCI is the world's largest shared CI/CD platform and the central hub for taking code from a concept to a deliverable.

3. Travis CI

Travis CI is a hosted CI service for building and testing software projects hosted on GitHub. Travis CI offers various paid and free open source plans for private projects. TravisPro is deployed on the customer's hardware and offers custom deployments of proprietary versions.

4. Concourse

Concourse is an automation system developed in Go, which is most commonly used in CI/CD to scale any automation pipeline (from the simple to the complex).

5. AWS CodePipeline

AWS CodePipeline is a fully managed CD service that helps you automate your release pipeline for fast and reliable updates to your applications and infrastructure.

Whenever code changes occur, CodePipeline automates the build, test and deployment phases of the release process based on your defined release model, allowing you to deliver features and updates quickly and reliably.

You can easily integrate AWS CodePipeline with third-party services (such as GitHub) or custom plugins. When using AWS CodePipeline, there are no upfront fees as you only pay for what you use.

6. Azure Pipelines

We build cloud-hosted pipelines for Linux, macOS and Windows. Build web, desktop and mobile applications and deploy them to the cloud or locally.

Spend less time on specifics and more time on creative things by automating builds and deployments through the pipeline.

VII. Continuous Deployment (CD) Tools

Why are CD tools essential for DevOps?

• Reducing pre-release stock.
• Automating complex pipelines.
• Unifying the team's goal of creating value for customers.

1. Spinnaker

Spinnaker is a free, open-source CD software platform originally developed by Netflix and later acquired and expanded by Google.

Spinnaker is a multi-cloud CD platform for the rapid release of software changes. It combines a powerful and flexible pipeline management system with the integration of major cloud providers.

2. Octopus Deploy

Octopus Deploy is an automated deployment and releases management tool that the world's leading CD teams use.

Octopus is a toolset that greatly simplifies the DevOps process for continuous testing and deployment of large numbers of microservices or applications via the cloud or virtual machines.

3. AWS CodeDeploy

AWS CodeDeploy is a fully managed deployment service that automates software deployments to various computing services such as Amazon EC2, AWS Fargate, AWS Lambda and local servers.

You can use AWS CodeDeploy to automate software deployments and reduce error-prone manual operations.

VIII. Configuration Management Tools

Why are configuration management tools essential for DevOps?

• Enforce consistency.
• Treat Infrastructure as codes.

1. Terraform

Terraform is an open source Infrastructure-as-Code (IaC) software tool developed by HashiCorp. Users can work with a high-level configuration language (called Hashicorp Configuration Language or JSON) to define and provide data centre infrastructure.

2. BOSH

BOSH is a project that unifies release engineering, deployment, and lifecycle management of small and large-scale cloud software. BOSH can provide and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime.

We developed BOSH to deploy cloud computing, but it can also be used to deploy any other software (e.g. Hadoop). BOSH is particularly suited to large distributed systems.

In addition, BOSH supports a variety of Infrastructure as a Service (IaaS) providers such as VMware vSphere, Google Cloud Platform, Amazon Web Services EC2, Microsoft Azure, OpenStack and Alibaba Cloud. BOSH provides a Cloud Provider Interface (CPI) that allows users to extend BOSH to support other IaaS providers such as Apache CloudStack and VirtualBox.

3. Chef

Chef is a configuration management tool for handling machine configuration on physical servers, virtual machines or in the cloud. Many companies use Chef to control and manage their Infrastructures, such as Facebook, Etsy, Cheezburger and Indiegogo.

Chef is a leader in continuous automation software, an innovator in application automation and one of the initiators of the DevOps movement. Chef works with over 1000 of the world's most innovative companies to provide practices and platforms for rapidly delivering software to achieve their digital transformation vision.

4. Ansible

Ansible is an open source software configuration, configuration management and application deployment tool. It runs on Unix family systems and can configure both Unix family systems and Windows, and it can use its declarative language to describe system configuration.

5. Puppet

Puppet provides the ability to define what software and configuration a system requires and then maintain the specified state after the initial setup.

Puppet uses a declarative domain-specific language (DSL) similar to Ruby to define configuration parameters for a particular environment or Infrastructure. Puppet discovers system information through a utility called Facter, which is installed along with the Puppet package.

The Puppet master node manages important configuration information for all nodes under its control through manifests.

Other nodes controlled by the master node have Puppet installed and run the Puppet agent (a daemon). The configuration information collected by the agent node about the node is sent to the master node. The master node compiles the directory according to how it should be configured, and the other nodes use this information to update their configurations.

Puppet uses a pull model where the agent node polls the host at regular intervals for site-specific and node-specific configurations. The Puppet proxy application typically runs as a background service on a managed node in this Infrastructure.

6. Google Cloud Deployment Manager

Google cloud deployment manager is an infrastructure management service that makes it easier to create, deploy and manage Google Cloud Platform resources.

IX. Cloud Platforms Tools

Why is a cloud platform critical to DevOps?

• Friendly automation.
• Runtime with observability.

1. Amazon Web Service

Amazon Web Service (AWS) is a secure cloud services platform that provides computing power, database storage, content delivery and other features enabling businesses to expand and grow.

In short, AWS can do all of the following：

• Running web and application servers and hosting dynamic websites in the cloud.
• Storing all files securely in the cloud so you can access them anywhere.
• Using hosted databases such as MySQL, PostgreSQL, Oracle or SQL Server to store information.
• Using content distribution networks (CDNs) to deliver static and dynamic files quickly around the world.
• Sending bulk emails to your customers.

2. Microsoft Azure

Microsoft Azure is a cloud computing service created by Microsoft to build, test, deploy and manage applications and services through Microsoft data centres.

Microsoft Azure, originally known as Windows Azure, is Microsoft's public cloud computing platform. It offers a range of cloud services, including computing, analysis, storage and networking. Users can choose from these services to develop and extend new applications or run existing applications in the public cloud.

3. Google Cloud Platform

Google Cloud Platform, provided by Google, is a set of cloud computing services used to run its end-user products (such as Google Search, Gmail and YouTube) on the same infrastructure.

Google Cloud Platform provides the computing resources needed to deploy applications on the web, providing a place for individuals and businesses to build and run the software, and connect the software users via the web.

We employ the cloud services provided by Google to meet business challenges, including data management, hybrid and multi-cloud, artificial intelligence and machine learning.

4. Pivotal Cloud Foundry

Pivotal Cloud Foundry (PCF) is a multi-cloud application platform (PaaS) managed by the Cloud Foundry Foundation, a 501 organization. It was originally developed by VMware and later transferred to Pivotal Software, Inc. and was brought back to VMware in late 2019 with the acquisition of Pivotal by VMware.

PCF is a multi-cloud platform for deploying, managing and continuously delivering applications, containers and features. PCF allows developers to rapidly deploy and deliver software without the need to manage the underlying infrastructure.

5. Heroku

Heroku is a cloud PaaS that supports multiple programming languages. one of the pioneering cloud platforms, Heroku, has been in development since June 2007, when it only supported the Ruby programming language but now supports Java Node.js, Scala, Clojure, Python, PHP, and Go.

Heroku is a container-based cloud PaaS. Developers use Heroku to deploy, manage and scale modern applications. The platform is elegant, flexible and easy to use, providing developers with the easiest way to distribute their applications.

6. Chinese Cloud Vendors

Ali Cloud, Tencent Cloud, Huawei Cloud, Jingdong Cloud and other vendors of various types are all available options for domestic users.

X. Container Schedulers Tools

The main task of the container scheduler is to start containers on the most appropriate host and connect them. It must handle failures through automatic failover, and it needs to be able to expand containers when a single instance has too much data to process/compute.

The three most popular container schedulers are Docker Swarm, Apache Mesos and Kubernetes.

1. Docker Swarm

Docker Swarm is a container scheduler developed by Docker. This clustering solution offers several advantages, such as the availability of the standard Docker API.

2. Apache Mesos

Mesos aims to build a scalable, efficient system that can support a large number of frameworks. This is a significant problem: some frameworks, such as Hadoop and MPI, are developed independently, thus making fine-grained sharing between frameworks impossible.

Mesos aims to add a resource-sharing layer that provides a familiar interface for frameworks to access cluster resources. Mesos delegates scheduling control to frameworks, as many already implement complex scheduling.

Frameworks fall into four categories based on the type of jobs running on the cluster, some of which provide native Docker support, such as Marathon. Support for Docker containers was added in Mesos 0.20.0.

3. Kubernetes

Kubernetes is a system for orchestrating Docker containers into logical units through the concept of labels and Pods. Pods are the main difference between Kubernetes and the other two solutions - they are a collection of containers that form a deployed and scheduled service. This approach simplifies the management of clusters compared to correlation-based scheduling of containers such as Swarm and Mesos.

The Kubernetes scheduler is tasked with monitoring Pods with PodSpecs, and NodeName specifies where to place the containers in the cluster.

This is a departure from Swarm and Mesos, where Kubernetes allows developers to bypass the scheduler by defining PodSpec and NodeName while running Pods.

The scheduler uses predicates and priorities to define which nodes the Pod should run on. We can override these defaults with a new scheduler policy configuration.

We can specify a JSON file via the command line flag policy-config-file, which will use the predicates and priorities described in this file when starting Kubernetes. The scheduler will use these defined policies.

XI. Monitoring & Logging Tools

Why are monitoring and logging tools essential to DevOps?

• Quick recovery.
• Responsiveness.
• Transparency.
• Reduced manual intervention in the event of an incident.

1. ELK

ELK is a collection of three open source products: Elasticsearch, Logstash and Kibana, all of which are developed, managed and maintained by Elastic.

• E means ElasticSearch: for storing logs.
• L means Logstash: for transferring, processing and storing logs.
• K means Kibana: a visualization tool (web interface).

2. Datadog

Datadog is a cloud-scale application monitoring service, servers, databases, tools, and services through a SaaS-based data analytics platform.

Datadog Application Performance Monitoring (APM or tracing) enables users to gain insight into application performance — from automatically generated dashboards that monitor key metrics such as request volume and latency to detailed trace information for individual requests — alongside logs and infrastructure.

When a request is made to an application, Datadog can visualize the distributed system trace and show the user system data about that request.

3. New Relic

New Relic is a San Francisco, California-based technology company that develops cloud-based software to facilitate website and application owners to track their services' performance.

New Relic's Application Performance Monitoring (APM) software analysis product provides real-time and trending data on web application performance and end-user experience satisfaction.

4. Prometheus

Prometheus is a free application for event monitoring and alerting. It records real-time metrics in a time-series database and supports flexible queries and real-time alerts based on an HTTP pull model.

The core principle of the Prometheus server is crawling, which means that the metrics endpoints exposed by the individual nodes are invoked. It collects these metrics periodically and stores them locally.

5. Zipkin

Zipkin is a distributed tracking system. It is used to collect the time data required to diagnose latency problems, which provides collection and lookup capabilities of this data.

You can focus directly on the specified location if a trace ID is available in the log file. Otherwise, you can query based on service, operation name, tags and duration attributes. It can aggregate some interesting data for you, such as the percentage of time spent on the service and whether the operation failed or not.

6. Azure Monitor

Azure Monitor provides a comprehensive solution for collecting, analyzing and executing telemetry data from the cloud and local environments to maximize the availability and performance of applications and services.

It can help you understand how your applications are performing and proactively identify the issues affecting them and the resources they depend on.

The following are a few scenarios where Azure Monitor can be used：

• Using Application Insights to detect problems between applications and dependencies.
• Correlating infrastructure issues with Azure Monitor for VMs and Azure Monitor for containers.
• Using log analysis to drill down into monitoring data for troubleshooting and in-depth diagnostics.
• Supporting large-scale operations with intelligent alerts and automated operations.
• Visualizing with Azure dashboards and workbooks.

XII. Communication Tools

Why are communication tools essential to DevOps?

• Connecting teams.
• Reducing waiting times.
• Improving team collaboration.

1. Slack

Slack is an instant messaging platform developed by the Slack Corporation. Essentially a chat room for the entire company, Slack is designed to replace email as the primary way to communicate and share. You can use channels for group discussions and private messaging for information, file sharing, etc.

2. Microsoft Teams

Microsoft Teams is a unified communication and collaboration platform that combines work discussions, video conferencing, file storage and application integration.

Teams is a chat-based collaboration tool that allows global, remote and distributed teams to collaborate and share information through a common space. You can use some cool features it offers, such as document collaboration, one-to-one chat, team chat, etc.

3. Google Hangouts

Google Hangouts is a communication software product developed by Google. Hangouts started as a feature of Google+ and in 2013 Google started to integrate the functionality of Google+ Messenger and Google Talk into Hangouts as a standalone product.

Hangouts Chat is an effective way to communicate within your organization. You can send messages to one or more colleagues, create chat rooms for multi-person discussions, and use bots to automate work. You can use Hangouts Chat in your computer browser and in your Android or iOS mobile applications.

4. Zoom

Zoom Video Communications is a company based in San Jose, California teleconferencing services. It offers a teleconferencing service that combines video conferencing, online meetings, chat and mobile collaboration. Skype is more suitable for teams looking for a holistic business solution, while Zoom is more suitable for teams that do frequent video chats and meetings. Neither tool, however, it built specifically for remote work.

5. Recommended Chinese Versions

Tencent Meeting, Ding Talk, FeiBook, WeLink, etc.

XIII. Knowledge-sharing Tools

Why are knowledge-sharing tools essential for DevOps?

• Reducing knowledge waste.
• Increasing the efficiency of new employees.
• Reduction in making the same mistakes.

1. GitHub Pages

GitHub Pages is a static site hosting service that takes HTML, CSS and JavaScript files directly from a repository on GitHub runs them during the build process and then publishes the site. You can find examples of GitHub Pages sites in the GitHub Pages examples collection.

2. Confluence

Confluence is a collaborative program developed and published by Atlassian, who developed Confluence in the Java programming language and first released it in 2004. Confluence is a Wiki collaboration tool to help teams collaborate and share knowledge effectively. You can use Confluence to capture project requirements, assign tasks to specific users, and manage multiple calendars at once with the Team Calendar plugin.

3. Jekyll

Jekyll is a static site generator with blogging support for personal, project or organizational sites. It was developed in Ruby by GitHub co-founder Tom Preston-Werner and is licensed under the MIT Open Source License. Jekyll is a parsing engine packaged as a Ruby gem for building static sites based on dynamic components.

4. Google Sites

Google Sites is a structured Wiki and web authoring tool provided by Google. The goal of Google Sites is to enable everyone to create simple websites that support collaboration between different editors. These sites are outstanding on every screen (from desktop to smartphone), and they can be done without the knowledge of design or program.

Conclusion

All the categories covered in this article can lead to better implementation of DevOps. Any of them are helpful, and I think all of them are what you need to assist teams in delivering value to customers in a sustainable way and to make you stand out.

I mentioned the DevOps toolchain, such as planning tools, issue tracking tools, source control management, building and testing code, continuous integration and deploying source code, managing configuration, using cloud platforms, and then monitoring and logging as communication and knowledge sharing.

I trust you have found this article interesting. DevOps is an exciting way to work, although it is not the only way or the best way. Better things may come in the future, but for now, it seems to represent a great way to work as a team. It gets teams working together to deliver value while also allowing them to focus on the customer, not just on technology or internal stuff, but also on speed, learning and the company's future.