| MC TestAutomation
|
| Subsystem
list
|
| Top
level
| | |
How to Read QAP Results Files
This page is a guide to reading and interpreting the contents of
QA Partner results files.
If you are involved in writing test scripts, you should also read
Standards & Guidelines for Results Files.
See also Formal Grammar.
Quick Summary
Most output lines can be identified by the first non-space characters on the line,
as shown below.
See example.
*
| Errors and warnings always begin with this character.
|
*** ERROR
| Indicates an application error (not a script error).
|
*** PROGRAMMING ERROR
| Indicates a script error (not an application error).
|
*** SCRIPT SETUP ERROR
| Indicates an error in the configuration of the local test system (user error).
|
*** EXCEPTION
| Indicates an exception raised by the script (not by the application).
|
@
| GUI action: reports what the script is doing to the application.
|
* WARNING
| Indicates something interesting that isn't an error.
|
Testcase
| a testcase.
|
ScriptEnter
| automatic setup that happens before the script proper begins.
|
=#= Functional Area:
| a section or sub-section that tests a particular feature.
|
~ Test Phase:
| Indicates the beginning of one of several test phases.
|
Screen Shot File:
| indicates that a screen shot has been taken (usually during error handling).
|
HACK
| Documents a hack in the script or infrastructure
that should be fixed someday.
|
<< subsystem >>
| debugging statement for script development.
|
Results Formats
Depending on where your results file came from, it may be in one of a number
of formats.
QA Partner Results File
A "raw" QAP results file can be read only with the QA Partner application.
It has the advantage that the indentation blocks are preceded by "+" or "-"
icons that you can click to "open" or "close" the indentation block.
Text Results File
QA Partner can save a results file as ASCII text.
The indentation blocks are indented using tabs,
and (obviously) can no longer be opened or closed on demand.
Depending on how the results were extracted, some information may be missing,
such as successful scripts and/or testcases, and statistics (summary info)
for scripts and/or testcases.
Printouts
QA Partner can print a results file directly.
In the resulting printout, indentation blocks are indented by only 2 spaces,
which makes it difficult to read.
The moral: don't print results directly from QA Partner.
Convert them to text first, then print the text file.
Another caveat: results files tend to be very wide.
We recommend printing in landscape using an 8-point monospace font (e.g. Courier New).
Overall Structure
Results files use indentation to indicate the heirarchical containment of
different sections of a results file.
When you view an original results file through the QAP application,
each indentation level has an "opener" gadget so you can show or hide its contents.
When a results file is converted to text, the "openers" are lost, but the
indentation is maintained.
Here is a summary of the overall structure of a results file, as defined by QA Partner:
- Suite - a suite is just a collection of scripts and suites.
Suites can contain other suites.
- Script (always present)
- Script Summary (always present) - includes timing and error counts
- Other Output Lines (optional)
- Testcase - there may be many of these
- Testcase Summary (always present for each testcase)
- includes timing and error counts
- Other Output Lines (optional)
- Other Output Lines (optional)
ScriptEnter
This section contains output generated by the error recovery system,
before the script proper ("main") begins execution.
It is during ScriptEnter that the application is normally launched,
if it wasn't running when the script started.
This section contains a list of any
Script Parameters
that were passed to the script, along with their values.
It contains the application name, version and model,
and the value of the seed for the QAP random number generator.
Thissection also contains a list of the source files used,
and information about the system the script was run on.
Testcases
Each testcase represents a single test, and is associated with
a particular functional area of the product under test.
Within the 4Test source code, a testcase is a special kind of subroutine.
Such a routine will often be called several times with different
parameters, performing a different test each time.
When this happens, the result file will contain a "disambiguator" line
inside each test case. This line contains text that distinguishes one
testcase invocation from another.
Disambiguator lines begin with a dagger character (looks sort of like '+').
Errors, Exceptions and Warnings
Errors are always preceded by three asterisks "***",
and come in four basic flavors which are described below.
All errors include a complete call stack that describes where in the
test script the error occured. This will normally be useful only to
someone familiar with 4Test (QAP) code.
All errors also include a context stack that describes the
indentation blocks within which the error occurred.
This information can also be obtained by looking backward through
the results file, but doing so can be tedious in a large results file,
which is why we include the context with each error.
Script Setup Errors
An error of this type indicates that you (the person running the script)
have made a mistake in how you ran the script.
This may be a problem in your
MClocal.inc
file, or it may be a problem with
Script Parameters.
Check the script documentation for Setup Procedures and Script Requirements.
The error message should make it reasonably clear what you need to fix;
if it doesn't, ask the owner of the script to improve the error message.
Programming Errors
An error of this type indicates a problem in the script.
You should contact the owner of the script, or any member of the
Test Automation Team, and let them know.
You should also enter a bug in ClearQuest against the script.
Exceptions
An exception statement in the results file indicates that an exception
was raised by the script (not by the application).
Exceptions are always considered errors, but it may indicate an error in the
script or an error in the application.
Further investigation is required to determine where the fault lies.
Exceptions are reported with a number and a description.
In most cases the description should provide all the information you need.
If you need to know what the number means, refer to
4Test Exception Codes and
List of Exception Codes.
Application Errors
An error that is labeled as "ERROR" (not one of the other types described above)
usually indicates that the script thinks it has found a problem in the application.
Warnings
Warnings begin with a single asterisk "*".
They are counted separately, and are not considered errors.
They are used to indicate something that isn't necessarily an error,
but may be of interest to the person running the script.
Product Features
The product feature under test is indicated by a line beginning with
"=#=", which is an indentation block.
All the results related to that feature are contained in that indentation block.
These feature blocks can be nested within other feature blocks.
Product features are identified by names which are carefully chosen so that
they can be manipulated through software. For a complete discussion, see
Product Features.
Note that the heirarchy of features in a results file may be different
from the official ETS feature heirarchy for the product.
Tools exist for mapping between these heirarchies.
Note that not all indentation blocks indicate features.
Features are always indicated by "=#=".
Test Phases
Lines beginning with "~ Test Phase:" indicate the beginning of
one of five possible test phases. These are:
- Setup
- During this phase, the script is setting up the application in preparation
for a test, but is not actually performing the test yet.
This typically includes opening projects and bins, loading sequences, etc.
If an error occurs during this phase, you cannot assume that it is directly related
to the feature currently under test.
This phase may be omitted; if it appears, it always precedes all other phases.
- Execute
- This phase contains the actual test itself, exclusive of any setup or cleanup.
This phase can only occur inside a testcase.
If an application error occurs during this phase, you can be reasonably sure
that the error is in whatever feature is currently under test.
- Verify
- During this phase, the test is verifying that the feature under test actually
did what it was supposed to do. This verification may be performed during the
execution phase, but some scripts may explicitly place it in its own phase for
clarity.
If an application error occurs during this phase, you can be reasonably sure
that the error is in whatever feature is currently under test.
This phase may be omitted; if it appears, it must not appear before Execute.
- Cleanup
- During this phase, the script is cleaning up after itself, after performing a test.
This phase may be performed even if an error occurred during execution or verification.
Cleanup may include deleting temporary files, closing windows, etc.
If an error occurs during this phase, you cannot assume that it is directly related
to the feature currently under test.
This phase may be omitted; if it appears, it must not appear before Execute.
- Results
- During this phase, the script produces information about the test.
Most scripts will not need to do this; typically only performance tests will
have data to report.
If an error occurs during this phase, you cannot assume that it is directly related
to the feature currently under test.
Screen Shots
When an exception terminates a testcase or script,
the error recovery system captures the entire screen in a bitmap
file that you can examine after the script is finished.
A script usually does not terminate as soon as it encounters an error;
it usually plods on and tries to run to completion.
Thus, when the script ends, the application is almost never in the state
it was in when the error occurred.
A screen shot can be extremely useful for figuring out what the application
was doing when the error happened.
Screen shot lines begin with "SCREEN BITMAP FILE", followed by the complete
path name of the screen shot file.
Screen shots (on Windows) are standard Windows bitmap files; there are a
variety of programs that will display them.
You can probably just double-click on the file and it will open.
The error recovery system takes screen shots for only the first N errors in each script.
You can set the value of N through the
iRECmaxScreenShotsPerScript
variable in your MClocal.inc file.
You can also specify where to store screen shots through the
glsCU_SCREEN_SHOT_DIR
variable.
Debugging Statements
Debugging statements begin with a double left angle bracket "«".
They are for use only during script development, and
should never appear in official production test runs.
A subsystem prefix (e.g. "REC", "MC", "EDL", "BIN", "PRJ") appears inside
the double angle brackets to identify which subsystem
the debugging statement belongs to.
Future Additions
The following line prefixes are not currently in common use,
but are being considered (and reserved!) for future enhancements.
Output File | gives the name of an auxiliary output file.
|
Summary | provides a summary of errors in the indentation block.
|
This page is maintained by
(REMOVED).
Last updated 17 May 2000.