Software Testing

Chapter 5: Software Testing

Acronyms

Acronym	Full Form
AI	Artificial Intelligence
API	Application Program Interface
ATDD	Acceptance Test-Driven Development
CMMI	Capability Maturity Model Integration
CSS	Cascading Style Sheets
DL	Deep Learning
EBSE	Evidence-Based Software Engineering
GUI	Graphical User Interface
HIL	Hardware-In-the-Loop
IoT	Internet of Things
ΚΡΙ	Key Performance Indicator
MC/DC	Modified Condition Decision Coverage
ML	Machine Learning
MTTR	Mean Time to Recovery
OAT	Orthogonal Array Testing
ODC	Orthogonal Defect Classification
SoS	System of Systems
SPI	Software Process Improvement
SPICE	Software Process Improvement and Capability Determination
SUT	System Under Test
TDD	Test-Driven Development
TMMi	Test Maturity Model integration
UI	User Interface
UP	Unified Process

Introduction

Software testing is the dynamic validation that a System Under Test (SUT) provides expected behaviors on a finite set of selected test cases. This activity is a crucial part of the software development life cycle, aimed at finding failures so that the underlying faults can be corrected.

Key terms in this definition include:

System Under Test (SUT): The object being tested, which can range from a single program or component to a complex System of Systems (SoS).
Test Case: A specification of inputs, execution conditions, testing procedures, and expected outcomes required to run a test. A collection of test cases is a test suite.
Dynamic: Testing requires the execution of the SUT, as opposed to static analysis techniques.
Finite: Because exhaustive testing is almost always impossible, testing uses a practical subset of all possible test cases. This implies a trade-off between limited resources and unlimited testing requirements.
Selected: The criteria for selecting test cases are critical. Techniques are chosen based on factors like risk, requirements, cost, and desired quality attributes.
Expected: For every test case, there must be a way to determine if the SUT’s observed outcome matches the expected one. This mechanism is known as a test oracle.

Software testing is a holistic activity that spans the entire development life cycle.

Breakdown of Topics for Software Testing

The Software Testing Knowledge Area (KA) is broken down into the following topics:

Software Testing Fundamentals
Test Levels
Test Techniques
Test-Related Measures
Test Process
Software Testing in the Development Processes and the Application Domains
Testing of and Testing Through Emerging Technologies
Software Testing Tools

1. Software Testing Fundamentals

1.1 Faults vs. Failures

It’s essential to distinguish between a fault (or defect) and a failure.

A fault is the underlying cause of a malfunction within the software.
A failure is an undesired effect or behavior observed in the system’s delivered service.

Testing reveals failures, but it is the faults that cause them that must be removed. Not all faults will manifest as failures.

1.2 Key Issues

Several key issues define the challenges and focus areas within software testing.

Test Oracle Problem: An oracle is any human or mechanical agent that decides whether a SUT behaved correctly for a given test. Automating oracles can be difficult and expensive, but they are essential for determining a “pass” or “fail” verdict.
Test Selection and Adequacy: A test selection criterion helps choose a subset of test cases, while an adequacy criterion helps determine when enough testing has been done.
Theoretical and Practical Limitations: As noted by Edsger Dijkstra, “program testing can be used to show the presence of bugs, but never to show their absence.” Complete testing is not feasible for any non-trivial software.
The Problem of Infeasible Paths: A control flow path that cannot be exercised by any input data is an infeasible path. Identifying these paths is important for path-based testing techniques to avoid wasting effort.
Testability: This refers to how easily a software artifact can be tested. It can mean the ease of satisfying a coverage criterion or the likelihood that a test suite will expose a fault if one exists.
Test Automation: A key challenge is to automate as much of the testing process as possible, from test case generation and execution to results analysis, to increase efficiency and coverage.
Scalability: This is the software’s ability to handle increasing loads, transactions, or data volumes. Testing must often account for complex, distributed, and virtualized environments.

1.3 Relationship of Testing to Other Activities

Software testing is related to several other software engineering activities, including:

Static analysis techniques
Quality assurance and improvement
Correctness proofs and formal verification
Debugging
Program construction
Security, effort estimation, and legal issues

2. Test Levels

Testing is performed at different levels, distinguished by the target of the test or its objective.

2.1 The Target of the Test

Testing is traditionally subdivided into four main stages or levels:

Unit Testing: Verifies the functionality of individual, separately testable software components, such as subprograms or modules, in isolation.
Integration Testing: Verifies the interactions among different software components. Common strategies include top-down, bottom-up, and “big bang” integration.
System Testing: Tests the behavior of the entire, integrated system against its specified requirements. It often includes testing non-functional requirements like performance, security, and reliability.
Acceptance Testing: Verifies that the SUT satisfies end-user needs and requirements. It is often conducted by or with end-users to ensure the software is fit for its intended purpose.

2.2 Objectives of Testing

Testing can be aimed at verifying different properties or achieving specific goals.

Conformance and Compliance Testing: Conformance testing verifies that the SUT meets specified standards or requirements, while compliance testing demonstrates adherence to laws or regulations.
Installation Testing: Verifies that the SUT is installed correctly in its target environment.
Alpha and Beta Testing: Alpha testing is trial use by a small group of potential users, while beta testing involves a larger set of representative users before the official release.
Regression Testing: The selective retesting of a SUT to verify that modifications have not caused unintended side effects and that the system still meets its requirements. This is a critical activity in Agile and DevOps environments.
Non-functional Testing: Validates non-functional aspects of the SUT. Types include:
- Performance Testing: Assesses characteristics like response time and capacity.
- Load Testing: Validates behavior under normal and peak load conditions.
- Stress Testing: Pushes the SUT beyond its specified operational capacity to see how it handles overload.
- Reliability Testing: Evaluates the SUT’s ability to perform its required functions under stated conditions for a specified period.
- Usability Testing: Evaluates how easy it is for end-users to learn and use the software.
- Compatibility Testing: Verifies that the software works with different hardware, software, or network environments.
Security Testing: Focuses on validating that the SUT is protected from external attacks and that its data has confidentiality, integrity, and availability.
Privacy Testing: Assesses the security and privacy of users’ personal data.

3. Test Techniques

Test techniques are systematic procedures for generating or selecting test suites. They are often classified based on the information source used to derive test cases.

3.1 Specification-Based (Black-Box) Techniques

These techniques derive test cases from the SUT’s specification or requirements without knowledge of its internal structure.

Equivalence Partitioning: Divides the input domain into classes of data from which test cases can be derived. The assumption is that one test case from each partition is as effective as any other.
Boundary Value Analysis: Selects test cases on or near the boundaries of input domains, as many faults tend to occur at these extremes.
Decision Table Testing: Uses tables to represent logical relationships between conditions (inputs) and actions (outputs), helping to systematically test all combinations.
State Transition Testing: Models the SUT as a finite-state machine and derives tests to cover its states and transitions.
Scenario-Based Testing: Uses models of the SUT’s behavior (e.g., use cases, user stories, workflow models) to generate test cases that validate its functionality from an end-to-end perspective.
Random Testing (Fuzzing): Generates test inputs randomly from the input domain. Fuzz testing, a modern application, uses random or unexpected inputs to find security loopholes and coding errors.

3.2 Structure-Based (White-Box) Techniques

These techniques use information about the SUT’s internal design or code to derive test cases.

Control Flow Testing: Aims to cover elements of the SUT’s control flow graph. Coverage criteria include:
- Statement Coverage: Execute every statement.
- Branch Coverage: Execute every branch (e.g., both true and false outcomes of an if statement).
- Path Coverage: Execute every possible path from entry to exit (often infeasible).
Data Flow Testing: Focuses on the points where variables are defined and used. It aims to cover paths between the definition of a variable and its subsequent uses.

3.3 Experience-Based Techniques

These techniques leverage the experience, intuition, and knowledge of human testers.

Error Guessing: Testers use their expertise to anticipate where faults are most likely to occur and design test cases to find them.
Exploratory Testing: Simultaneous learning, test design, and test execution. Testers dynamically design and execute tests based on their observations of the SUT’s behavior.
Ad Hoc Testing: Less formal techniques that rely on a software engineer’s skill and intuition. Examples include monkey testing (random inputs) and pair testing (two people working together).

3.4 Fault-Based and Mutation Techniques

Fault-Based Testing: Devises test cases specifically to reveal predefined categories of faults.
Mutation Testing: A small, syntactic change is made to the SUT to create a “mutant.” The goal is to create test cases that can “kill” the mutant by distinguishing its behavior from the original program. It is often used to evaluate the quality of a test suite.

Measures are needed to evaluate the SUT and the effectiveness of the testing process itself.

4.1 Evaluation of the SUT

Fault Statistics: Classifying found faults and tracking their frequency (fault density) can help predict quality and improve the development process.
Reliability Evaluation: Statistical estimates of software reliability, often derived from observing failures during testing, can be used to decide when the SUT is ready for release.
Reliability Growth Models: These models use failure data collected during testing to predict future reliability as faults are fixed.

4.2 Evaluation of the Tests Performed

Coverage Measures: The percentage of items covered by a test suite, according to a specific criterion (e.g., 90% branch coverage).
Fault Injection: Intentionally introducing faults into the SUT to see if the test suite can find them.
Mutation Score: In mutation testing, the score is the ratio of killed mutants to the total number of generated mutants. A higher score indicates a more effective test suite.

5. Test Process

Testing concepts and techniques must be integrated into a defined and controlled process.

5.1 Practical Considerations

Attitudes: A collaborative, “egoless” attitude towards finding and correcting failures is crucial for successful testing.
Test Planning: Testing activities must be planned, including defining objectives, coordinating personnel, identifying resources, and managing risks.
Test Documentation: Documentation is essential and includes test plans, test specifications, test designs, and incident reports.
Test Team: The organization of the test team can vary, from developers testing their own code to independent, dedicated test teams.
Test Completion Criteria: A decision must be made about when to stop testing. This decision is based on coverage measures, fault density, reliability estimates, and risk analysis.

5.2 Test Sub-Processes and Activities

Test Design and Implementation: Creating and specifying test cases based on the chosen techniques.
Test Environment Setup: Preparing the necessary hardware, software, and tools to conduct the testing.
Test Execution: Running the test cases according to documented procedures and recording the results.
Test Incident Reporting: Documenting unexpected outcomes and anomalies. This data is the basis for debugging and fixing problems.

5.3 Staffing

Staffing involves defining roles and responsibilities for testing activities, identifying hiring needs, and defining training requirements to ensure the team has the necessary skills.

6. Software Testing in Development Processes and the Application Domains

6.1 Testing Inside Software Development Processes

Traditional Processes (e.g., Waterfall): Testing is often treated as a distinct phase that occurs after development is complete.
Shift-Left Processes (e.g., Agile, DevOps): Testing is integrated throughout the life cycle, starting as early as possible. Test-Driven Development (TDD) and continuous integration rely heavily on automated regression testing to provide rapid feedback.

6.2 Testing in the Application Domains

Testing approaches are often tailored to the specific needs of an application domain.

Automotive: Involves extensive security, reliability, and simulation testing of embedded hardware and software.
Internet of Things (IoT): Focuses on security, privacy, interoperability, and performance across a heterogeneous network of devices.
Healthcare: Must ensure compliance with regulations (e.g., HIPAA), interoperability between systems (e.g., HL7, FHIR), privacy, and safety.
Mobile: Requires testing for usability, compatibility across many devices and operating systems, and performance under varying network conditions.
Finance: Demands high levels of security, reliability, performance, and compliance with financial regulations.

7. Testing of and Testing Through Emerging Technologies

7.1 Testing of Emerging Technologies

New technologies present unique testing challenges.

AI/ML/DL Systems: Testing is difficult due to their non-deterministic nature. It involves validating the data, the learning algorithm, and the resulting model for correctness, robustness, and fairness.
Blockchain: Requires stress testing for transaction volume, penetration testing for security, and property testing for smart contract logic.
Cloud Applications: Focuses on performance, scalability, elasticity, security, and compatibility in a distributed, virtualized environment.

7.2 Testing Through Emerging Technologies

Emerging technologies can also be used to improve the testing process.

Machine Learning: Can be used to optimize test case design, prioritization, and evaluation, as well as to automate bug triaging.
Cloud Computing: Provides scalable, on-demand infrastructure for large-scale performance testing, simulations, and running massive test suites in parallel.
Simulation: Allows for the evaluation of SUT execution under critical or hard-to-replicate conditions. Hardware-in-the-loop (HIL) simulation is common in embedded systems.
Crowdsourcing (Crowdtesting): Leverages a large, diverse group of testers to validate an application on a wide variety of devices and real-world conditions, especially for mobile apps.

8. Software Testing Tools

Tools can significantly improve the efficiency and effectiveness of testing by automating labor-intensive tasks.

8.1 Testing Tool Support and Selection

Selecting the right tools is critical and depends on factors like the development process, SUT technology, and testing objectives. Often, a suite of tools is needed to cover all needs.

8.2 Categories of Tools

Test Harnesses (Drivers and Stubs): Provide a controlled environment to execute parts of a SUT in isolation.
Test Generators: Assist in the creation of test cases, whether randomly, from a model, or based on code paths.
Capture/Replay Tools: Record test interactions and replay them to automate test execution, especially for GUI testing.
Coverage Analyzers: Measure how much of the SUT has been exercised by a test suite according to a given criterion (e.g., statement or branch coverage).
Regression Testing Tools: Support the re-execution of test suites and can help select which tests to run after a change.
Load and Performance Testing Tools: Generate heavy loads of users or transactions to evaluate SUT performance.
Security Testing Tools: Include tools for penetration testing, vulnerability scanning, and fuzz testing.
Test Management Tools: Help manage the entire test process, including planning, test case management, execution tracking, and reporting.