APIs

Subsystem
list

Top
level

Standards & Guidelines for Results Files

This page defines standards and guidelines for QAP results files.

The target audience is script writers and maintainers of infrastructure.

General Philosophy

Writing a Test Script

Maintaining Infrastructure

Test Execution

General Philosophy

Results files should attempt to attain the following goals:

Provide enough information about failures so that they can be reproduced by hand.
Clearly distinguish between application errors, script errors, and user (setup) errors.
Provide the information needed by the Test Automation Database in a format that can be easily interpreted by a post-processing program.
Minimize the learning curve for reading results files.
Not be so incredibly big that they're practically unusable.

Every line in a results file should fall into one of the following categories:

Errors, warnings and exceptions
GUI gestures (what is the script doing to the application)
Feature identification (what product features are under test)
Program structure (to break up lengthy output and make it more readable)
Output data (not all scripts will need this)
Debugging (should only be used during script development)

Writing Test Scripts

If you are writing a test script, you are responsible for providing the following information in the results file:

Errors and warnings for conditions explicitly checked by your script.
The features that are being tested.
Testcase disambiguators, where needed.
In some cases, the various test phases.
Output data, if any.
Debugging info.

In general, your script should not log the following information:

Exceptions (see below).
How the script is manipulating the application (CUlogGuiAction). This is handled for you by the infrastructure.

Reporting Errors

Never use the built-in 4Test routine LogError!

When your script discovers an error condition, it should log the error in the results file in one of a number of different ways, depending on whose fault the error is:

Application
Errors These are errors in the application. If your script or testcase can reasonably continue executing after discovering an error, then report it using this routine:
void CUlogError (string sErrorMessage);
If your script or testcase cannot reasonably continue executing after discovering an error, then report it by raising an exception with a 4Test raise statement.
Programming
Errors These are errors made by the script writer (you), not by an application developer. Check for them using Assertions.
Setup
Errors These are error made by the person running your script. Check for them and/or log them using these routines:
void RequireSetup (boolean bCondition, string sExplanation)
void CUsetupError (string sExplanation)

Composing Error Strings

Regardless of what mechanism you use to report an error, you should observe certain rules when you compose your error string:

Do not begin your error message with "I'm an error" indicators such as "ERROR", a line of asterisks, etc. The various error reporting utilities take care of that for you in a consistent way. Just say what the error is, and don't use any "funny characters" (see Comments & Test Structure below).
Include sufficient data in the message so that the person running the script can figure out what's going on. An error message like "can't open file" is useless. Instead, say comething like "can't open file FOO.BAR because it doesn't exist". If a received value differs from an expected value, include both values in the error string. You can use the "+" operator to concatenate strings, and you can cast any data type to a string using [string]. You can also embed the values of variables in a string by enclosing the variable name in curly braces { }.

Warnings

The use of warnings is reserved for indicating the use of obsolete or deprecated programming constructs. Warnings indicate a problem with a script (i.e. a programming error) which needs to be corrected eventually, but is not serious enough to prevent the script from being deployed.

Logging Exceptions

The only time your script should explicitly log an exception is when it explicitly catches the exception and does not reraise it. In all other cases, you should let the underlying infrastructure log the exception. Otherwise, the exception will be logged twice, leading to confusion and unnecessary clutter in the results file. In other words, if you catch an exception, you can reraise it or log it, but not both. (You can also recover from the exception and print nothing.)

If you must log an exception, always use the following routine:

void CUlogException (string sAdditionalInfo optional);

Output Data

Here we are referring to data that is generated directly by the script (i.e. through Print statements), not data that is generated by the application (i.e. bins, clips, EDLs, media files, etc.).

Output data must be clearly distinguished from other lines in the results file in one of the following ways:

Begin each line of output data with a sequence of characters that will distinguish it from other output lines, such as "Data". (But don't use reserved characters.) If you have different kinds of data, you can mark it with different words. This approach works best for very small amounts of data.
Put your output data inside its own indentation block, using the built-in 4Test functions ResOpenList and ResCloseList. Provide a string to ResOpenList that will distinguish the indentation block from other lines in the file, such as "Data". Don't put anything other than output data in this block. This approach works best for moderate amounts of data.
If you have a large amount of output data, put it in a separate file. If the file contains information that should be stored in the Test Automation database, log the name of the file by calling the following routine:
void CUlogOutputFileName (string sFileName, string sDescription optional)

For more information on test data, read this.

Functional Areas

Your script must report the functional areas that it tests. You must do this inside each testcase by calling CUtestcaseFeature (see below). If it is convenient, you can block out sections of your test that test a particular functional area by calling the following routines:

CUlogBlockState CUbeginFeatureTest (CUfeatureOrList)
void CUendFeatureTest (CUfeatureOrList)

Example #1

In this example, the functional areas are completely defined inside the testcases.

testcase tcTestDrop ()
{
    CUtestcaseFeature({"Digitize","Functions","Batch Digitize","Drop"});

    ... body of test ...

}

testcase tcTestNonDrop ()
{
    CUtestcaseFeature({"Digitize","Functions","Batch Digitize","Non-Drop"});

    ... body of test ...

}

Example #2

In this example, the functional areas are partially defined inside the testcases, and are relative to the functional area defined in main.

testcase tcTestDrop ()
{
    CUtestcaseFeature("Drop");

    ... body of test ...

}

testcase tcTestNonDrop ()
{
    CUtestcaseFeature("Non-Drop");

    ... body of test ...

}

main ()
{
    CUbeginFeatureTest({"Digitize","Functions","Batch Digitize"});

    tcTestDrop();
    tcTestNonDrop();

    CUendFeatureTest({"Digitize","Functions","Batch Digitize"});
}

For more information on features and functional areas, click here.

Test Phases

A results file must properly report the phases of each test. The various test phases are:

1. eCU_TEST_SETUP Perform any setup that the test requires, such as opening bins, loading sequences, changing settings, etc. Some tests may not require any setup; some or all of a test's setup may be performed implicitly by the recovery system.
2. eCU_TEST_EXECUTE This phase should only include the operation being tested. Typically, this phase will consist of a single call to a subroutine or method. Every test must include this phase.
3. eCU_TEST_VERIFY Verify the results of the test, to ensure that the test was successful. Some verification may be performed implicitly by the infrastructure and/or the recovery system.
4. eCU_TEST_CLEANUP Perform any cleanup that the test requires, such as deleting sequences, closing dialogs, etc. You should normally leave around anything that the test created, so that it can be examined after the test script finishes running. Most tests will not require this phase.
5. eCU_TEST_RESULTS Display any results of the test. Most scripts will not use this phase; it is intended for scripts which first perform a set of tests, gathering data as they go along, and then print all the gathered data at the end.

Some of this reporting is taken care of for you by the Recovery System. Here is a summary:

ScriptEnter (the beginning of a script) sets the test phase to eCU_TEST_SETUP.
When a testcase begins execution, the test phase is implicitly set to eCU_TEST_SETUP.
When a testcase finishes execution, the test phase is implicitly set to eCU_TEST_CLEANUP.
When you call CUbeginFeatureTest, it sets the test phase to eCU_TEST_SETUP.
When you call CUendFeatureTest, it sets the test phase to eCU_TEST_CLEANUP.

In each testcase, you must explicitly set the test phase to eCU_TEST_EXECUTE immediately before executing the actual test. You may also set other test phase as needed. To set the test phase, call this routine:

void CUlogTestPhase (CUtestPhase ePhase);

Testcases

Within each testcase, you must report the functional area under test by calling CUtestcaseFeature. You should call it near the top of the body of your testcase.

Within each testcase, you must indicate the beginning of the actual test (as opposed to setup for the test) by calling CUlogTestPhase. If your testcase performs an cleanup after executing the test, you must indicate that with another call to CUlogTestPhase.

If your testcase takes parameters, then it must log information that distinguishes between different invocations of the testcase. Do this by placing a call to CUdisambiguateTC near the top of the body of the testcase.

Comments and Test Structure

It's a good idea to print occasional lines that describe, at a high level, what your test is trying to do. There are no special API's for such output; simply use the built-in 4Test functions Print and Printf.

You can begin any line with a period; such lines will never be interpreted to have special meaning.

Debugging Info

At some time during script development, you will probably want to print some information to the log file to help you debug the script. You should print such information by calling the following routine:

void CUdebugPrint (string sSubSystemPrefix, varargs of anytype);

You need not explicitly test a debugging flag before calling this routine; that functionality is built into CUdebugPrint.

Reserved Characters

Do not begin any lines of output with any of the following sequences of characters. They have special meaning to tools that read and interpret results files. You may use these sequences within a line, as long as they are not the first characters on the line.

* asterisk
- hyphen
< less than
> greater than
+ plus
: colon
[ open bracket
] close bracket
( open paren.
) close paren.
/ forward slash
\ backslash
! exclamation
@ at
# sharp
$ dollar
% percent
^ circumflex
& ampersand
_ underscore
= equals
? question mark
Script
Suite
Testcase

A line beginning with a period will never be interpreted to have special meaning.

Maintaining Infrastructure

The methods and routines of the infrastructure are responsible for the following:

Performing a reasonable amount of error checking. You should check potential error conditions that are the most obvious, or the easiest to check. Don't try to check everything, or perform checks that will take a long time to execute. When you find an error, raise an exception (don't log it).
Reporting GUI actions.
Reporting test phases.
Reporting the build and testbed names.

Reporting GUI Actions

Report GUI actions by calling the following routine:

void CUlogGuiAction (string sAction, window w);

The action string should begin with a present-tense verb such as "click", "type", "select", or "close", and end in a preposition such as "in", "on", "to", or "from". If the verb takes both a direct and an indirect object, make sure that the one that isn't the window is included in the action string. For example, when reporting the selection of a menu item from a popup menu, include the name (or number) of the item in the action string.

The window should be the window, control, or pane that the action is being performed on. When calling CUlogGuiAction from a method, the window will almost always be "this". CUlogGuiAction automatically reports the window's parents.

Debugging Info

Print debugging information by calling the following routine:

void CUdebugPrint (string sSubSystemPrefix, varargs of anytype);

You need not explicitly test a debugging flag before calling this routine; that functionality is built into CUdebugPrint.

Test Execution

A few pieces of information must be provided by the person who is executing the test.

The name of the testbed must be provided in the LocalConstants.inc file by defining the following variable:

const string gsMC_TESTBED = "Lab System #1 (Moe)";

The internal release name of the product under test must be provided through the standard script parameter sMCinternalReleaseName. Examples are "Twister", "Jaguar", "Whirlpool".

This page is maintained by (REMOVED).

Last updated 12 May 2000.

Application Errors	These are errors in the application. If your script or testcase can reasonably continue executing after discovering an error, then report it using this routine: `void CUlogError (string sErrorMessage);` If your script or testcase cannot reasonably continue executing after discovering an error, then report it by raising an exception with a 4Test `raise` statement.
Programming Errors	These are errors made by the script writer (you), not by an application developer. Check for them using Assertions.
Setup Errors	These are error made by the person running your script. Check for them and/or log them using these routines: `void RequireSetup (boolean bCondition, string sExplanation)` `void CUsetupError (string sExplanation)`

1.	`eCU_TEST_SETUP`	Perform any setup that the test requires, such as opening bins, loading sequences, changing settings, etc. Some tests may not require any setup; some or all of a test's setup may be performed implicitly by the recovery system.
2.	`eCU_TEST_EXECUTE`	This phase should only include the operation being tested. Typically, this phase will consist of a single call to a subroutine or method. Every test must include this phase.
3.	`eCU_TEST_VERIFY`	Verify the results of the test, to ensure that the test was successful. Some verification may be performed implicitly by the infrastructure and/or the recovery system.
4.	`eCU_TEST_CLEANUP`	Perform any cleanup that the test requires, such as deleting sequences, closing dialogs, etc. You should normally leave around anything that the test created, so that it can be examined after the test script finishes running. Most tests will not require this phase.
5.	`eCU_TEST_RESULTS`	Display any results of the test. Most scripts will not use this phase; it is intended for scripts which first perform a set of tests, gathering data as they go along, and then print all the gathered data at the end.