Automated test frameworks

Whether in automated testing practice or in daily communication, we often hear one word: framework. Many people have always been aware of the word "framework" but don't know the meaning. Today let's talk about something about automation frameworks.

1. What is the framework

A framework is a box - referring to its constraining, as well as a shelf - referring to its supportability. It is a basic conceptual structure used to solve or deal with complex problems.

In software engineering, framework is the reusable design of the whole or part of the system, which is expressed as a set of abstract components and the method of interaction between component instances.

Another definition holds that a framework is an application skeleton that can be customized by application developers.

The former is defined in terms of applications and the latter in terms of purpose.

The above concept is a little bit abstract, let's explain from another view:

1. The framework itself is generally not complete enough to solve a specific problem;
2. Frameworks are born designed for expansion;
3. The framework can provide many convenient and easy-to-use tools for subsequent expansion components. In other words, A framework is a set of libraries or tools that help solve a certain type of problem.

Source: Freepik

Constraints: A software framework for solving a specific problem will first define the boundary of the problem, and then constrain the relevant software components within this boundary, maintaining the cohesion of the framework in solving the problem.

Supportability: The framework itself does not solve any problems but provides a combined foundation to the relevant components to solve the problem. The scientificity and ease of use of this foundation directly affect the scientificity and convenience of further development on top of it.

The above two explanations and definitions of the framework do not seem to be very simple and straightforward to explain what a framework is. Below is my personal understanding:

• Definition: A tool that constrains the boundary to solve some specific problem, supports the entire problem solution, and is composed of some problem-solving components.
• Specific Questions: What's the problem? -- Automated test
• Constraint Boundaries: Why make a constraint? -- Clarify the scope and purpose of the test
• Solution: What is the solution to the problem? -- Programming language + Tools + Other
• The components that make up the tool: which components? -- Use case, Script, Data, Log, Report, Notification
• Tools: What are the features? -- Flexibility, scalability, high cohesion and low coupling

2. Automated test

2.1 Why make an automated test?

1. Black box testing regression is inefficient
2. The contingency and uncertainty of manual testing
3. Insufficient coverage of regression
4. The quality of the delivered product cannot be guaranteed, it depends on the evaluation
5. The more complex the system, the more problems
6. Butterfly effect caused by long online time and high component failure rate (fast iteration, more overtime work)

Source: Freepik

2.2 What problems can be solved by Automated tests?

1. Improve the response rate after a problem occurs
2. Reduce return costs
3. Improve regression coverage
4. Improve return efficiency
5. Improve regression stability

2.3 Disadvantages of Automated test

1. It is impossible to reduce the cost investment but to speed up the feedback of test results and improve test quality
2. Automation is suitable for regression instead of discovering bugs
3. The recording and playback function is tasteless, and visualization is not a good practice
4. Not all functions of all systems are suitable for automated testing

3. Automated testing framework

The components that make up the framework should at least have the following functions:

• Log: log recording and management functions, set different log levels for different situations, which is convenient for locating problems;
• Report: Test report generation and management and instant notification, quick response to test results;
• Source: Management of configuration files and static resources, following the principle of high cohesion and low coupling;
• Common: Management of public functions, methods, and common operations, following the principle of high cohesion and low coupling;
• TestCase: Test case management function, one function point corresponds to one or more cases, to improve the coverage rate as much as possible;
• TestData: Test data management function, separate data and scripts, reduce maintenance costs and improve portability;
• TestSuite: Test component management function, assemble and build different test frameworks for different scenarios and different needs, and follow the flexibility and extensibility of the framework;
• Statistics: Test result statistics management function. Statistics, analysis, comparison and feedback of the results of each execution test, data-driven, provide references for software optimization and process improvement;
• Continuous: Continuous integration environment, also called CI environment, including test file submission, scan and compilation, test execution, report generation and timely notification, etc. Continuous integration is the core of automated testing!

Source: Freepik

4. Common automated testing frameworks

4.1 Interface Automation Framework:

1. java+testNG/Junit+Maven/Ant/Gradle+Jenkins+MySQL+testlink/redmine
2. python+unittest/pytest+Git+Jenkins+MySQL+testlink/redmine
3. python+rebotframework+unittest/pytest+Git+Jenkins+MySQL+testlink/redmine
4. jmeter+Maven/Ant+Jenkins+MySQL+testlink/redmine

4.2 UI Automation Testing Framework

1. java+selenium/appium+testNG/Junit+Maven/Ant/Gradle+Jenkins+MySQL+testlink/redmine
2. python+selenium/appium+unittest/pytest+Git+Jenkins+MySQL+testlink/redmine
3. python+rebot framework+unittest/pytest+Git+Jenkins+MySQL+testlink/redmine

What do you see from the common frameworks above?

They all have common features: Programming Language + Unit Testing Framework + Scan and Compilation tool +Continuous Integration tool + Database + Project Management tool.

Source: Freepik

• Programming language: writing test scripts, logging and output;
• Unit testing framework: Provides a series of configurations such as test script running, exception checking, etc.;
• Scanning and compiling tools: Scanning and compiling test files, generally being used with the continuous integration tools to get a better result;
• Continuous integration tool: Jenkins -- a classic continuous integration tool;
• Database: test data management;
• Project management tools: statistical management of test results;

There are too many automated testing tools. The above only lists some open source tools that are widely used. The specific framework selection needs to be determined according to the characteristics of the specific project and the technical characteristics of the team and individual!