Guidelines for Using Test Data

This document discusses some issues regarding the use of test data, and offers some guidelines and standards for same.

Where Should Input Data Live?
Where Should Output Data Live?
Standard Projects and Bins
The Default Project and Bin

Where Should Input Data Live?

By "Input Data", we mean data that is used as an input to a piece of test code. We assume that without its input data, such test code is unusable. Therefore, maintaining the integrity of the input data is just as important as maintaining the integrity of the test script code. And so, test data should be stored under source control (ClearCase). However, this does not adequately answer the question of where it should live.

There are basic choices of where test data can reside: within a 4Test file (part of the test script), or in an external file. (Data can also be generated on the fly at runtime, but such data is out of the scope of this discussion, and ultimately is generated from other data, even if the starting point is merely a seed for a random number generator.) Here are some pros and cons regarding these choices:

Data Lives in Test Script Data Lives in External File
The test script must be modified in order to modify the test data. The test data can be modified independently of the test script.
The syntax of the test data can be checked at compile time. The syntax of the test data must be checked at run time.
The data is used primarily by the test script, and is not read directly by the application. The data may be read by the test script or by the application (or both).

Data Lives in Test Script	Data Lives in External File
The test script must be modified in order to modify the test data.	The test data can be modified independently of the test script.
The syntax of the test data can be checked at compile time.	The syntax of the test data must be checked at run time.
The data is used primarily by the test script, and is not read directly by the application.	The data may be read by the test script or by the application (or both).

Some types of data, such as Projects, Bins and Media Files, cannot reasonably be stored in a 4Test file, and must reside in files external to the test script. Furthermore, some such files must live in locations dictated by the operating system or the application.

A Standard Location for External Input Data

In MClocal.inc there is a constant called glsMC_INPUT_DATA_DIR that contains a pathname to a directory in which all external input data files should reside. This path will normally point into the ClearCase directory "/ppg/QAP_Canned_Data". This directory contains a sub-directory for each subsystem whose scripts require external input data. Input data files that are used by a variety of subsystems live in a directory called "Shared".

Golden Samples

A "golden sample" is an output file that has been examined and determined to satisfy some set of requirements. It is used for comparing against a future output file to make sure that the new output file is correct. Golden samples are used when it is not feasible for a test script to examine an output file and analyze it for correctness.

Note that while golden samples were originally output files, for the purposes of a test script, they function as input files: they are read by the script (or some auxiliary comparison program), not generated by the script, or by the application during script execution. Therefore, golden samples should be stored along with other external input data.

Media Files

Because of their size, we do not currently encourage checking them into ClearCase. A solution to this issue needs to be found.

Where Should Output Data Live?

By "Output Data", we mean data that is generated by the test script or the application as a result of executing the script.

If output data is generated directly by the test script, there are two basic choices of where it can reside: within the QA Partner results file (using Print, printf, etc.), or in a separate file.

If output data is generated by the application, the application may (or may not) dictate where the data resides.

A Standard Location for Output Data

In MClocal.inc there is a constant called glsMC_OUTPUT_DATA_DIR that contains a pathname to a directory in which all output data files should reside, except for the QA Partner results file, and files whose location is dictated by the application.

The output directory should contain a sub-directory for each subsystem whose scripts generate output data files. That sub-directory should be created by the scripts as needed.

What About Results from Previous Test Runs?

The user of a test script may (or may not) want to retain the results of previous test runs. The designer of a test script has two choices:

Overwrite the old results each time the script is run. If the user wants to retain old results, he must copy them elsewhere before running the script again. However, the user does not have to worry about discarding old unwanted results.
Encode the date and time in the name of the output file, or place the output file a directory whose name includes the date & time (this directory should live inside the subsystem's output directory). The user does not have do take any special action to retain old test results, but he is responsible for periodically discarding any old results that are not longer needed.

Golden Samples

Golden samples are treated as input files. Read this.

Standard Projects and Bins

Standard Data Main Page

Additional notes are here.

The Default Project and Bin

In MClocal.inc there is a pair of constants called gsMC_DEFAULT_PROJECT and gsMC_DEFAULT_BIN. These constants define a project and bin which may be used by any test script.

Requirements for Content

The default project is required to contain at least one bin (which must be the default bin), and the bin is required to contain at least one simple source clip. There is no guarantee that there is media associated with any clips. The default project and bin may or may not be one of the standard test projects and bins.

Rules for Usage

If a test script uses the default project or bin, it can make no assumptions about their contents other than what is stated above. In particular, you should not assume that the default project is one of the standard projects, nor should you assume that the default bin is one of the standard bins.

This page is maintained by (REMOVED).

Last updated 29 January 1998.