DevSecOps Security Practices (Testing)

Preface

With the development of DevOps, DevOps has dramatically increased the speed of enterprise application iteration. But at the same time, security that fails to keep pace not only counteracts the enhancements brought about by the DevOps change and slows down the digital transformation process of enterprises but also leads to vulnerabilities and risks unannounced. In 2012, Gartner introduced the concept of DevSecOps, which organically integrates security protection processes into the traditional DevOps process to provide strong assurance of development security and support the implementation of development.

The importance of security testing

Software testing is an essential part of the software development lifecycle and is an important component. There are various forms of testing, and according to the type of testing, they can be divided into interface-type testing, functional testing, performance testing, documentation testing, etc. Traditional functional and non-functional testing can implicitly find certain vulnerabilities in a program in some form, such as password format validation. Still, these tests are not specifically looking for security flaws and are not comprehensive enough to ensure that an application or service has passed a comprehensive security vulnerability test. Security testing can test how software is handled in the event of an attack/malicious damage by an unauthorized internal or external user and ensure that software and data are secure.

In Forrester's 2021 survey statistics, "The State of Application Security, 2021", Forrester Research, March 23, it was revealed that of the 480 identified external attacks on global enterprises, web applications were at the top of the list at 39%, with attacks on web applications mainly referring to program-based SQL injections, cross-site Web applications are mainly program-based SQL input, cross-site scripting or remote file inclusion attacks. Attacks on software vulnerabilities accounted for 30 per cent of the attacks, mainly referring to the exploitation of security vulnerabilities.

Top 10 reported ways external attacks were carried out, according to Forrester “The State of Application Security, 2021” report

Security testing is a key part of a DevSecOps practice, where software programs are tested in various ways to ensure quality. Security testing should not only involve software programs. Still, it should look at end-to-end pipelines, real-time production systems, software infrastructure, databases and middleware to reduce the risk of security attacks. Security testing is a specialist area with tools and practices designed to expose these vulnerabilities.

How to do security test well ?

How to do security testing well, let's focus on three aspects of application security testing AST, software composition analysis SCA and penetration testing.

1. Application Security Testing - AST

As seen in the previous statistics on attack targets, applications remain the main target, acting as a gateway between customers and core business functions. Customers use applications to access services the business provides or purchase products, which provides the greatest security risk to the business and its customers. At the same time, with the presence of a global epidemic, the rise of telecommuting forms of work has made enterprises more reliant on applications. Hence, application security is vital, and application security testing also bears the brunt of security testing. Today's techniques commonly used in the industry are divided into three main categories: static application security testing SAST, dynamic application security testing DAST, and interactive application security testing IAST.

Image Source: Mend

Static Application Security Testing（SAST)

Static Application Security Testing is a testing method that finds security vulnerabilities in application code by examining the application's source code. Some tools rely on the compilation process or even binaries to improve coverage and accuracy through abstract syntax trees, control flow analysis and taint tracing. SAST is a white-box testing approach with common tools such as Coverity, Checkmarx, FindBugs, CodeQL, and ShiftLeft inspect.

SAST tools have excellent code coverage and can find more and more types of vulnerabilities in the code. Using SAST can be a low-cost defence against common security flaws hidden in the source code and open vulnerabilities such as OWASP Top 10. The problem can be pinpointed to specific lines of code for easy fixing. However, with SAST, there is a problem with false positives, which can reduce the tool's usefulness, and engineers may need to spend more time removing false positives rather than vulnerabilities. Still, most SAST tools provide a way for engineers to configure the scanner to reduce the number of false positives.

Dynamic Application Security Testing (DAST)

Dynamic Application Security Testing (DAST) is the process of constructing specific inputs to an application by simulating the behavior of an attacker without the need for system source code and analyzing the application's behavior and reactions to determine if the application has certain types of security vulnerabilities. Common tools include AWVS for commercial and open-source web applications and some for the computer or endpoint AppScan.

DAST replicates real attacks from an attacker's perspective and, if successful, proves the exploitability of vulnerabilities in an application, so the number of false positives is much lower than SAST, the accuracy is very high, and engineers can focus on fixing real vulnerabilities rather than spending time verifying them. DAST is a reliable vulnerability detection technique that can find many real security vulnerabilities. DAST uses attack signature libraries to do vulnerability discovery and validation and can find the most high-risk issues, making it a very common security testing solution used by the industry for web security testing.

DAST is a form of black-box testing; during DAST testing, the main focus is on functionality and no programming skills are required to understand the internal logic structure of the application. Fuzzy testing is a common approach in DAST testing, where program failures are induced by deliberately introducing incorrectly formatted or random data into the application. Fuzzy testing strategies are developed based on the application's intended use and the application's functional and design specifications.

DAST can scan for vulnerabilities in third-party open-source components, third-party frameworks, and the application itself. Many DAST tools can work across multiple languages and frameworks, providing greater application coverage than SAST tools. But this comes at a price: it is often difficult to pinpoint where to apply fixes in the source code when vulnerabilities are found.

Configuring DAST requires advanced security knowledge to properly set up tests based on the potential attack surface of the target application. Engineers must be able to configure dynamic testing tools to get the most accurate results. They also need to integrate it into the application at runtime, where it fits better into the CI/CD pipeline and where runtime analysis can be performed. Stuart Gunter speaks highly of DAST as an automated testing tool. Its low false alarm output and the action close to a manual penetration test can produce excellent results when used properly.

Interactive Application Security Testing (IAST)

Interactive Application Security Testing (IAST), introduced by Gartner in 2012, is a runtime inspection tool that relies on the activity of an application while monitoring the traffic flowing through an application to determine if the underlying source code is vulnerable to exploitation. IAST monitors data flow from a running application to determine if security controls are in place to protect the data. For example, it will detect if a string passes the authentication method or if a password is encrypted at the source and remains encrypted until it exits the running application. Because it looks at the behavior of data at runtime rather than making assumptions about how individual components manage data locally, IAST is rarely a false alarm. At the same time, IAST tools can read and understand the underlying source code, so they can locate and report lines of code that programmers should be concerned about to fix vulnerabilities identified by the scan.

Despite its simplicity of use, IAST has the same limitations as DAST. It requires a running application to execute the test, best integrated into the CI/CD stream. And if the functional tests executed during testing do not cover 100% of the underlying source code, the security test code coverage will also be less than 100%.

There are three approaches to automated application security testing: static, dynamic and interactive, all of which play a role in a successful DevSecOps framework, and the decision on which to integrate into the value stream depends on several factors. SAST is most likely to have flaws found in the source code, but due to the lack of context, it generates a lot of false positives, and supplementing SAST with DAST or IAST (or both) is a good solution.

Image Source: Mend

2. Software Composition Analysis - SCA

One of the reasons why applications are a major target of external attacks is the increasing use of open-source software. To speed up development, developers reuse a lot of proven components, libraries and other codes, and organizations use open-source and third-party components, which exposes more APIs. The organization's code base contains codes that the developer did not write. It is unknown if it has been tested for security or contains malicious codes or malware. Worse still, many of these open-source libraries rely on other open-source libraries to provide additional functionality. This means there is a chain of OSS dependencies in the organization's applications. According to Synopsys, open source makes up 70% of the application code base, meaning that the organization's engineers do not write the majority of an organization's applications. The challenge is understanding if these dependencies are secure and, if not, how to ensure they are.

Image Source: Devops School

While application security testing can identify some vulnerabilities independent of code, it is unlikely to identify all associated issues. Known vulnerabilities may be recorded in generic vulnerability databases, and some open-source providers maintain their vulnerability databases. It is important to check application dependencies against these databases and address any risks associated with vulnerable components. Several software portfolio analysis tools are available to perform this task, each with its advantages and disadvantages. At a basic level, they all provide detailed information on vulnerabilities and recommendations for risk reduction, such as upgrading to a secure version of a component or using a different component altogether.

Usually, a secondary SCA is performed during the security testing phase, the first time being at an early stage of software development. This is because it is cheaper to update third-party libraries during development than in a production environment, where fixes are more expensive.

3. Penetration testing

The AST application testing and SCA software component analysis described above are both automated during the development of security practices. Although there are many layers of automated testing in the value stream, it cannot be 100% secure. Manual security testing is essential as an effective complement to automated testing. In some regulated industry sectors, an accredited third party must conduct penetration testing of the organization's online products or services by an accredited third party. Typically these penetration tests are carried out in a pre-production environment. They can also be carried out on live systems.

The National Cyber Security Centre (NCSC) defines a penetration test as "an authorized test of a computer network or system designed to find security vulnerabilities". Penetration testing is an effective method of testing where security personnel simulate a hacker's access to a system. It is very demanding on the person carrying out the test, as a competent tester will be able to find valuable security vulnerabilities. In contrast, a tester without strong attack capabilities will not be able to effectively find security vulnerabilities in a system, making it difficult to use on a large scale. The process of penetration testing is not repeatable (it relies heavily on the experience of the tester, similar to debugging), so the idea of scripted, automated penetration testing is not feasible and fully automated tools can usually only find specific security vulnerabilities that can be discovered in a standard way, such as simple SQL injection. During penetration testing, authorized testers attempt to use the same techniques and tools used by attackers, exploit known vulnerabilities and common misconfigurations, conduct penetration tests against system architecture, application, and network-level vulnerabilities, and implement penetration tests based on industry characteristics and business scenarios. The scope should cover important security risk points and critical business systems with a clear penetration testing plan and management mechanism.

Image Source: Imperva

In DevOps, there may be multiple releases in a day. Therefore, it is not feasible to run manual tests before the system is deployed. Instead, manual penetration testing is conducted as part of an annual review to meet regulatory requirements and as a validation process to highlight weaknesses in the automated security testing process.

Example: Huawei Cloud Security Development Practice

Huawei Cloud ensures the security of the verification phase from three main aspects: specification, method and tools.

• Specify certain coding specifications for each language, test design specifications, guidelines and testing guidance for each test, technical specifications for secure compilation options, and Fuzz security testing guidance.
• MMethodologies, following security compilation options, static analysis, SCA, TMBT security test design, Web security testing techniques, generic component security testing, virtualization security testing methods, IoT security testing, virus scanning, Fuzz.
• In terms of tools, Huawei Cloud has developed its own proprietary tool platform by taking customer security requirements and industry standards as check items, SecSolar Secure Coding Checking Service, SecGuard Security Testing Service, SecFuzz, and SecureCat can check the security configuration of the industry's mainstream OS and DB.

Image Source: Huawei Cloud

All Huawei Cloud services are subject to multiple rounds of security testing before release. In the Alpha stage, microservice-level functional and interface security tests such as authentication and session security are conducted. In the Beta stage, service integration is verified through fuzzing tests of APIs and protocols, and in the Gamma stage, security-specific tests such as database security are conducted. The test cases cover the security requirements identified in the security design phase and penetration test cases from the attacker's perspective.

At the end

When it comes to security testing differs from traditional manual testing methods, with automation being a core requirement wherever possible. DevOps engineers can deliver features quickly with minimal security risk by automating security testing in the continuous integration pipeline. Feedback loops from automated testing allow engineers to identify and fix features before they appear in production. When combined with an effective education policy and good design principles, security test automation enhances continuous learning through these feedback loops. It provides an environment where security is at the heart of software delivery without negatively impacting processes and customer outcomes.