Identifying Software Test Architect Skills and Knowledge

I. Introduction

IEEE and ISO define the concept of software testing architecture. An important aspect of standardizing the software test architecture concept is knowing the skills required for those who do this work. These people are called software test architects in industry and this paper.

Modern systems include a large amount of software that introduces lots of risks. Test teams often fail to understand the complete picture of the system and develop test plans and strategies that are not as in-depth and comprehensive as they could be. In addition, the testing industry lack awareness of the concept of architecture and the challenges that engineers face in defining software test architectures and test environments. The lack of understanding of test architectures leads to an increased potential for systems to fail once released, as well as facing many challenges in terms of security, reliability, interoperability, functionality, usage, etc.

Some companies create the " software test architect" position. Others may refer to these individuals as systems or software engineers but require them to possess the knowledge and skills to build software test architectures and environments. Recently there have been more than 2,000 job vacancies for software test architects on www.linkedin.com. Certification is available specifically for this title. However, there is still a lack of consensus across the industry for this role. This article not only describes the skills and basic knowledge that a software test architect should be equipped with but also includes reflections on these required skills, intending to provide a reference for the work on the ISO/IEEE/IEC 29119 software testing series of standards.

The context of this article considers the situation where a complex system driven by software may require an architect. The responsibilities of the test architect for complex software systems are: to develop appropriate test plans, test strategies, test architectures, and test environments.

II. Where Is A Software Test Architect Needed?

Not every software testing project needs a test architect. Many software systems with a simple test architecture have been included in the test plan and strategy. For example, in a low-risk PC program running on a single machine, we only need to write simple manual test cases without the support of tools. Many systems may resemble this situation; however, the number of complex software systems is increasing, thus creating a requirement for software test architects. Complex software systems that require a software test architecture include：

• Artificial intelligence systems with sophisticated data analysis capabilities.
• Extensive data analysis systems for many sites/providers.
• Virtual reality and gaming systems with numerous users.
• Cloud computing systems.
• A mobile system with millions of devices.
• Embedded systems.
• IoT network.
• Embedded connectivity systems (e.g. modern cars)

Any complex software systems with internal and external software system interactions will likely require some degree of Software Test Architecture (STA). This involves dedicated personnel responsible for designing the software test architecture early in the testing process and during strategy development and their work for the entire software development process. In Figure 1, each complex system contains a software architecture to develop test plans and define the architecture. In addition, every region of the world, the aerospace, consumers, cities, and other elements can be interconnected via the cloud. These worldwide interconnected systems are best evaluated and tested in a super STA. The complexity of software and testing architectures increases with the number of software elements, risks, and connections. The complexity of software and test architectures increases with the number of software elements, risks, and connections.

III. Software Test Architect Responsibilities

Definition of Software Test Architect: The Software Test Architect is a professional who uses strategies, methods, practices, processes, and tactics to develop software test architectures. On this basis, the responsibilities of a test architect include:

• A senior test technologist develops appropriate test strategies, technical approaches, and automated test plans to safeguard the system's overall quality throughout the software lifecycle.
• Management that can deliver products and projects successfully.
• Keep up to date with the latest methods and techniques in the software testing industry and deliver them to other engineers as required throughout the product lifecycle.
• Define the software architecture of the test system, including test environment, infrastructure, and automation. Note: The test system includes the environment, tools, and operations to support the testing.
• Build, monitor, and maintain optimum test systems, architectures, and techniques.
• Work with roles and stakeholders in other project areas (e.g. hardware, software, O&M, etc.) to ensure proper integration.
• Maintain communication with management on test plans, status, etc.
• Identify architectural requirements and limitations of test systems to improve processes and select test automation methods, data, and tools as required.
• Document and track risks for the test architecture.
• Create test system models, including various views and layers as required.
• Delineate the logical units of the test system by defining, validating and maintaining the test architecture.
• Work with testers to define and explain the test architecture and the design philosophy of the test system.

Software test architects carry the same authority and responsibility as system architects, hardware, and network architects.

IV. Direct Experience and Skill Set

A software test architect should be a senior professional with years of training and a wide range of knowledge and skills. Their knowledge and skills in the product area cover testing, software, systems, and other engineering areas. Specifically, these architects need to be proficient in the following knowledge and skills:

1. Business understanding of requirements, architecture and development abilities, skills in testing and quality, communication capabilities, and leadership.

2. Close cooperation with engineering departments, management and stakeholders.

3. Training and guidance in test planning, strategy, and architecture.

4. The ability of the cost and schedule estimates.

5. The knowledge and planning skills in the product area include：

   • An understanding of the software system under test.
   • Partners and suppliers of support products.
   • Test management, standards, certification and guidance.
   • The development of test and project process improvement objectives.
   • An understanding of the software development lifecycle.
   • Testing / V&V methods and techniques.
   • Test design, execution and post-analysis.
   • Test architecture.
   • The test environment includes hardware, software, tools, operations, simulation, techniques, components, users, prototypes, databases, analysis, and other applicable elements.
   • Advanced or appropriate methods are used to develop test plans and test strategies.
   • Test automation for efficiently implementing project plans and achieving test objectives.
   • Testing and quality programs.
   • Development team processes and plans.
   • A global view of engineering and testing.
   • Software integration, deployment and operation.

V. Ancillary Knowledge and Skill Areas

In addition to knowledge and skills related to software testing, test architects need to know and be able to practice skills from other engineering disciplines, such as：

1. Hardware

2. Software

3. Systems

   • Operations & Deployment

4. Management, social skills, leadership and communication

5. Development and processes

   • System and software requirements
   • Design (incl. testability)
   • Implementation techniques
   • Quality guarantee
   • Configuration management (for hardware and software)

6. Security

   • Design for security
   • Encryption and data transmission
   • Penetration Testing

7. Quality features

   • Maintainability
   • Performances
   • Interoperability & Integration
   • Reliability
   • Other essential quality characteristics required for the system

8. Identification of potential project/test risks and impacts

9. Documentation and technical communication

These are followed by several 'non-engineering and hard-to-quantify key concepts, which include: artistic, aesthetic, and human factors aspects that make techies and their skills stand out.

Then, critical thinking is an essential complementary skill for test architects. It is necessary to use various skills to contemplate and progress from an abstract STA plan to concrete design details and then to the final test results. The architect must start with the whole "picture" of the system, using various skills to achieve an elegant and optimal STA solution, and then refine this into a test design. A creative test architect knows and practices many skill areas and is adept at using a variety of patterns appropriate to the system (example in Figure 1). Test architects need to consider different negative test scenarios that may be overlooked by system and software architects. As a result, the STA may be more complex than the original system under test.

VI. Evidential Support and Future Work

The information in sections 2 and 3 comes from the author's 20 years of experience working on complex systems of this type. The information is also derived from staff who are working on ISO 29119. The project that gave us this experience originated from the aerospace industry, where the author retired after 35 years of working in the software industry. He worked as a software test architect for several complex systems. Moreover, some of the knowledge is based on information and research articles available on the Internet.

This article is intended to be a starting point for a discussion on developing standards and a system of knowledge for software test architects. The required information is incomplete, and we expect to receive it from other fields, architects, and testers.

VII. Summary

The research in this article shows that more work is required in IoT testing architectures and environments. The skills and knowledge of a software test architect are different but complementary to those of the following roles.

• Test Manager: focuses on budget, personnel, plans, and logistics.
• Quality Manager: Focuses on process quality, validation, and review across lifecycle and software/system releases.
• Testers: focus on the day-to-day testing process, technology selection, execution and analysis.

Not every engineer can or should become a test architect, just as not everyone should become a manager. However, we must acquire knowledge and skills in the full spectrum of testing. Engineers who can support test plan development and process/product improvement may, with practice, become excellent architects. People who work as software test architects need to develop a breadth and depth of knowledge in all areas of software engineering.