Validation & Verification

Tom Kelliher, CS 319

Sept. 25, 1998

Announcements

From Last Time

Project requirements discussion.

Outline

1. The testing process.

2. Test Planning.

3. Testing Strategies.

Assignment

Read Chapter 5.

Introduction

• Validation: Are we building the right product?

• Verification: Are we building the product right?

• The difference: Consider a recipe for chocolate chip cookies. We may follow the recipe (verification), but not care for the result because we don't like walnuts (validation).

Terminology:

1. Static V & V: structured reviews of system representations (documents, source code). Formal methods. Can be applied to all stages of the process.

2. Dynamic V & V: exercising an implementation. Only applies to prototype or final system. The predominant method.

3. Statistical testing: Select tests which reflect typical usage patterns. Calculate system reliability, performance.

4. Defect testing: Select tests which reflect the requirements specification. Uncover deficiencies in the software.

5. Testing vs. debugging.

6. Regression testing: after fixing a defect, re-run the test suite.

The Testing Process

Stages of the testing process:

1. Unit testing: Independent testing of units.

2. Module testing: A module is a set of dependent components, such as a class or set of dependent functions. Can be tested independently of other modules.

3. Sub-system testing: Sub-systems may be independently designed and implemented. Predominantly interface errors. Integrate modules into sub-systems and test. Push on the interfaces.

4. System testing: Integrate sub-systems. Erroneous interactions between sub-systems and system. Validate against requirements.

5. Acceptance testing: System tested with procurer supplied data, rather than synthesized data. May reveal defects in requirements spec. Additional requirements validation. Also known as alpha test.

Beta test used for generic software: deliver an early version to prospective customers. Some vendors have elevated this to an art form.

Test Planning

Testing is expensive and time consuming. Plan for slippages.

Components of a test plan:

1. The testing process: overview of the test process.

2. Requirements traceability: how are the requirements being validated?

3. Test items: which software process products are being tested?

4. Testing schedule: overall testing schedule and resource allocation.

5. Test recording procedures: record results, allow for audits.

6. Hardware and software requirements: SW tools and hardware utilization.

7. Constraints: anticipate any possible testing constraints.

The relationship between software process, test plans, and testing:

Testing Strategies

Mix and match strategies. Test incrementally.

1. Top-Down testing:
   1. Start with most abstract components. Use with top-down design.

   2. Early validation, prototype. Psychological, managerial advantages.

   3. Use of stubs. How to generate returned items?

   4. Upper levels may not produce (testable) output.

2. Bottom-Up testing:
   1. Start at the bottom. Use with object-oriented design, reusable components.

   2. Architectural flaws may not be discovered until late. May force re-write of implemented components.

   3. Test drivers can be provided along with system to support reuse.

3. Thread testing:
   1. Test the thread/communication pathways within a system, particularly real-time, event driven system.

   2. Sequence: single event, multiple events of single type, multiple events of various types. Incremental approach again.

4. Stress testing:
   1. Determine if system can handle its intended load (and beyond!).

   2. Determine system failure behavior.

   3. May uncover hidden flaws.

   4. Extremely useful for distributed systems.

5. Back-to-Back testing: Test different versions of a system against each other. Diff the results.

Exercises

1. How do you test?

2. Is validation difficult?

3. Why is top-down testing not useful for object-oriented systems?

4. Why is regression testing necessary?

5. Can a programmer objectively test their own code?

6. Consider the ethics of testing until you exhaust the test budget, then deliver the product.