Validation & Verification

Tom Kelliher, CS 319

Sept. 25, 1998


From Last Time

Project requirements discussion.


  1. The testing process.

  2. Test Planning.

  3. Testing Strategies.


Read Chapter 5.



  1. Static V & V: structured reviews of system representations (documents, source code). Formal methods. Can be applied to all stages of the process.

  2. Dynamic V & V: exercising an implementation. Only applies to prototype or final system. The predominant method.

  3. Statistical testing: Select tests which reflect typical usage patterns. Calculate system reliability, performance.

  4. Defect testing: Select tests which reflect the requirements specification. Uncover deficiencies in the software.

  5. Testing vs. debugging.

  6. Regression testing: after fixing a defect, re-run the test suite.

The Testing Process

Stages of the testing process:

  1. Unit testing: Independent testing of units.

  2. Module testing: A module is a set of dependent components, such as a class or set of dependent functions. Can be tested independently of other modules.

  3. Sub-system testing: Sub-systems may be independently designed and implemented. Predominantly interface errors. Integrate modules into sub-systems and test. Push on the interfaces.

  4. System testing: Integrate sub-systems. Erroneous interactions between sub-systems and system. Validate against requirements.

  5. Acceptance testing: System tested with procurer supplied data, rather than synthesized data. May reveal defects in requirements spec. Additional requirements validation. Also known as alpha test.

Beta test used for generic software: deliver an early version to prospective customers. Some vendors have elevated this to an art form.

Test Planning

Testing is expensive and time consuming. Plan for slippages.

Components of a test plan:

  1. The testing process: overview of the test process.

  2. Requirements traceability: how are the requirements being validated?

  3. Test items: which software process products are being tested?

  4. Testing schedule: overall testing schedule and resource allocation.

  5. Test recording procedures: record results, allow for audits.

  6. Hardware and software requirements: SW tools and hardware utilization.

  7. Constraints: anticipate any possible testing constraints.

The relationship between software process, test plans, and testing:

Testing Strategies

Mix and match strategies. Test incrementally.

  1. Top-Down testing:
    1. Start with most abstract components. Use with top-down design.

    2. Early validation, prototype. Psychological, managerial advantages.

    3. Use of stubs. How to generate returned items?

    4. Upper levels may not produce (testable) output.

  2. Bottom-Up testing:
    1. Start at the bottom. Use with object-oriented design, reusable components.

    2. Architectural flaws may not be discovered until late. May force re-write of implemented components.

    3. Test drivers can be provided along with system to support reuse.

  3. Thread testing:
    1. Test the thread/communication pathways within a system, particularly real-time, event driven system.

    2. Sequence: single event, multiple events of single type, multiple events of various types. Incremental approach again.

  4. Stress testing:
    1. Determine if system can handle its intended load (and beyond!).

    2. Determine system failure behavior.

    3. May uncover hidden flaws.

    4. Extremely useful for distributed systems.

  5. Back-to-Back testing: Test different versions of a system against each other. Diff the results.


  1. How do you test?

  2. Is validation difficult?

  3. Why is top-down testing not useful for object-oriented systems?

  4. Why is regression testing necessary?

  5. Can a programmer objectively test their own code?

  6. Consider the ethics of testing until you exhaust the test budget, then deliver the product.

Thomas P. Kelliher
Wed Sep 23 17:11:05 EDT 1998
Tom Kelliher