TSY&A - Test Step Yield & Analysis - LEGACY

LEGACY - Use the new Test step analysis for improved analytics

The Test Step Yield & Analysis report lets you investigate your top failing and most time-consuming steps as well as visual representations of the actual measurements. By applying the desired values in the report filter, you will be presented with a grid of software-versions that matches your filter. Select the software versions you want to investigate, by selecting checking the checkbox in the grid, and press Apply Selected to load the report.

Top 10 - Step execution time

A chart display top 10 steps that consume most execution time. By default, the chart is ordered by average step execution time. You can also select to order by maximum/minimum execution time.
By selecting the option Display Sequence Calls, you can see the main, and sub-sequences with their respective total time-usage.

Top 10 failures 

The graph will display top 10 failed steps (with status equal to Failed, Error or Terminated). The chart also has a new summary “step” named Other. This bar represent the rest of the step failures. By clicking any of the bars in this chart you will be taken to the Step Details View of the step you select.

To the right, you can choose to hide or show the included steps in this chart. The Display sequence calls option, lets you view this chart based on total failures for each sequence call.

The Only count failures cause UUT failure option is only valid when this flag is set on a step in the UUT report. Typically, NI TestStand will only set this flag when a step has status Failed (not Error or Terminated), and when the Step Failure Causes Sequence Failure option is enabled (default). 

Step Grouping

Since this report potentially contains test steps from multiple different software runs, this chart, and the list of steps is based on all these software files merged together. The options for Step Grouping allows you to control how this merge is done. By default the option in set to Name, Id and Group which will display all steps, with these parameters matching, as the same step.

When changing the options for this chart you need to click Apply to apply your changes.

The Step List

The rest of the report contains a list off all steps included in the report, displayed as a merged test sequence. Here you can see, for each step, how many times is has been run and failed, as well as min, max and average step-time. You can sort the step list based on different criteria, by using the icons in the Step name column header.

Displaying Measurement Details

From this list, you can enter the Step Details View to investigate measurements, either by clicking the step-name (display measurement for this step only), or clicking the Open Details (displays measurement from all selected steps). By default you will se that all the steps included in the Top 10 failures chart are checked by default.

Measurement details for multiple steps

Although it is some times useful to investigate the measurements of multiple steps simultaneously, and WATS lets you look at up to 20 steps at a time, this is rarely practical. We advice that you investigate the data for individual steps, or a few at a time when steps are related, comparable and suspected of affecting each other.

See the article TSY&A - Step Details for more information on the Step Details view.