An article on the whole process of CI / CD pipelines

Starting with the CI / CD process, a series of steps containing all stages and responsible for creating automated and seamless software delivery is called the CI / CD pipeline workflow. Using the CI / CD pipeline, software release artifacts can move and advance in the pipeline from the code submission phase to the testing, construction, deployment, and production phases. The concept is powerful because once a pipeline is specified, part or all of it can be automated, accelerating the process and reducing errors. In other words, the CI / CD pipeline makes it easier for businesses to deliver the software multiple times a day automatically.

DevOps engineers are often confused with the CI / CD pipeline for automation at various stages in the CI / CD. While different tools can automate the various complex stages in CI / CD, the entire software supply chain of CI / CD may still be broken due to human intervention. So, first, understand the CI / CD process stages and why the CI / CD pipeline is critical for organizations to deliver code quickly and at a large scale. 

CI / CD phase: Understand personnel, processes, and technology

The enterprise application development team comprises developers, tester / QA engineers, operational engineers, SRE (Site Reliability Engineer), or IT operations teams. They work closely to deliver high-quality software to their customers. The CI / CD combines two separate processes: continuous integration and continuous deployment. The main steps are listed below.

Source: zhuanlan.zhihu.com

CI continuous integration

Continuous integration (CI) is building software and completing initial testing. Continuous deployment (CD) combines code with infrastructure, ensures that all testing is completed and policies are followed, and then deploys the code to the intended environment. Of course, many companies have their processes, but the main steps are as follows.

CI: Code submission

Source: zhuanlan.zhihu.com

Personnel: Developer and Engineer, Database Administrator (DBA), Infrastructure

Team Technology: GitHub, Gitlab, BitBucket

Process: The code submission phase is also known as Version Control Systems. Submission is the operation to send the latest changes written by the developer to the repository. Each version of the code written by the developer is stored indefinitely. After discussing and reviewing the changes with the collaborators, the developer will write the code and submit it after the software requirements, feature enhancements, bug repair, or change requests are completed. The repository that manages to edit and submit changes is source Management (SCM tool). After the developer submits the code (a request), the code changes are merged into the essential code branch stored in the central repository (such as GitHub).

CI: Static code analysis

Personnel: Developer and Engineer, Database Administrator (DBA), Infrastructure Team, Tester

Technology: GitHub, Gitlab, BitBucket

Process: Once the developer writes the code and submits it to the repository, the system automatically triggers and starts the following code analysis process. Imagine a step where the submitted code builds directly but fails during the build or deployment process. In terms of resource utilization, it is a slow and expensive, both machine and human, process. The static policy of the code must be checked. SAST (Static Application Security Test): SAST is a white-box test method that uses SAST tools such as SonarQube, Veracode, and Appscan to check the code internally to discover software defects, vulnerabilities, and weaknesses (such as SQL injection, etc.). This is a quick check process of checking the code for syntax errors. Although this phase lacks the functionality to check for run-time errors, this will be performed at a later stage.

Putting additional policy checks into an automated pipeline significantly reduces the number of errors found later in the process.

CI: Build

Source: zhuanlan.zhihu.com

Personnel: Developer and Engineer

Technology: Jenkins, Bamboo CI, Circle CI, Travis CI, Maven, Azure DevOps

Process: The goal of the continuous integration process is to accept regular code submission and continuously build binary artifacts. The continuous integration process helps discover bugs faster by checking if the added modules fit well with existing modules. This helps to reduce the time to verify the new code changes. Build tool helps compile and create executable or packages (. exe,.dll,.jar, et al.) depending on the programming language used to write the source code. SQL scripts are also generated during construction and then tested with the infrastructure profile. In short, the build phase is the phase of compiling the application. Other sub-activities of the build process include artifact storage, build validation, and unit testing.

CI: Testing stage

Source: zhuanlan.zhihu.com

Personnel: Tester and QA Engineer

Technology: Selenium, Appium, Jmeter, SOAP UI, Tarantula

Process: Release a build process A series of automated tests to verify the accuracy of the code. This stage helps to prevent errors from reaching the product. This check can last from seconds to hours, depending on the size of the build. For large organizations that submit and build code by multiple teams, these checks will run in parallel environments to save valuable time and notify Bug to developers as early as possible.

These automated tests are built by testers (or QA engineers), who have built test cases and scenarios based on user stories. They performed regression analysis stress testing to examine deviations from the expected output. The activities involved in the test are fitness testing, integration testing, and stress testing. This is a very advanced level of testing. Here problems will be found that the developers may not know.

Integration testing:

Integration testing is performed using tools like Cucumber, Selenium, where the individual application modules are combined and tested as a group while assessing compliance with the specified functional requirements. After integration testing, approval is required to move the update set in that group to the next stage, usually a performance test. This validation process can be cumbersome, but it is an integral part of the whole process. Some new solutions have emerged during the verification process.

Load and pressure test:

Load balancing and stress tests are also performed using automated test tools (e. g., Selenium, JMeter, etc.) to check whether the application is stable and performs well in high-flow environments. This test usually does not run on each update as the comprehensive stress test is extended. When significant new features are released, multiple updates are grouped, and complete performance tests are completed. In cases where a single update is transferred to the next stage, the pipeline may include the Canary test as an alternative.

Continuous deployment: bake and deployment

Source: zhuanlan.zhihu.com

Personnel: Infrastructure Engineer, Field Reliability Engineer (SRE), Operations Engineer

Technology: Spinnaker, Argo CD, Tekton CD

Process: After the test phase is completed, the cleared standard code can be deployed to the server, where it will be integrated with the main application. Before being deployed to the production environment, they are deployed to a test/staging or beta environment used within the product team. Before moving the builds to these environments, the build must go through two sub-phases, Bake and Deploy. Both phases are inherent to the Spinnaker.

CD: Bake

Bake is creating an immutable image instance from the source code with the current configuration in the production environment. These configurations may be something like database changes and other infrastructure updates. Spinnaker can trigger Jenkins to perform this task, and some organizations prefer to use Packer.

CD: Arrange

Spinnaker automatically passes the baked image to the deployment phase. This is setting the server group to the deployment to the cluster. Similar to the above test procedure, the same functional process is performed during the deployment phase. The deployment moves first to the testing stage, finally to the production environment, and then to approval and inspection. Tools like Spinnaker handle the whole process.

CD: Test and verify

This is also the key to optimizing the entire CI / CD process. Because many tests have been conducted now, failures should be rare. But any fault needs to be resolved as soon as possible to minimize the impact on the end customer. Teams should also consider automating this part of the process.

Deployment to the production environment is performed using deployment policies (such as blue-green deployment, Canary analysis, scrolling updates, etc.). The running application is monitored during the deployment phase to verify that the current deployment is correct or that a rollback is required.

CD: Supervisory control

Personnel: SRE, O & M Team

Technology: Zabbix, Nagios, Prometheus, Elastic Search, Splunk, Appdynamics, Tivoli

Process: To make a software distribution failure-secure and robust, it is crucial to track the health of the distribution in the production environment. The application monitoring tool tracks performance metrics such as CPU utilization and releases latency. The log analyzer will scan the log stream generated by the underlying middle ware and the operating system to identify the behavior and track the source of the problem. Notice any problems during production to ensure the safety and reliability of the production environment. In addition, the monitoring phase helps companies gather information about how new software changes contribute to revenue and helps infrastructure teams track system behavior trends and conduct capacity planning.

Continuous deployment: feedback and collaboration tools

Source: zhuanlan.zhihu.com

Personnel: SRE, Ops, and Maintenance Team

Technology: Zen Tao, ServiceNow, Slack, Email, Hipchat

DevOps teams aim to release more quickly and continuously and reduce errors and performance issues. This is achieved by frequent slack or email feedback to developers and project managers on new version quality and performance and timely ticket increases in the ITSM tool. Generally, the feedback system is part of the entire software delivery process. Therefore, any changes in the delivery process are frequently recorded into the system so that the delivery team can take action on it.

Businesses must evaluate an overall continuous delivery solution that can automate or facilitate the automation of the above stages.