Quality Assurance of massive LabVIEW SCADA System

Background

LabVIEW is programmed in a graphical environment, and this more or less requires a review to be graphical as well. You cannot simply say that just because the code is ugly it doesn’t work, but you can offer guidance on how to improve the code for readability and performance.

One challenge with a review like this is that the system has already grown so much that it will be hard to do any major changes, just because of the sheer number of code modules that may be affected. Another challenge is that the amount of code to go through is more than you can do in a limited period of time.

The primary goal of the review was to:

• Find potential problems to ensure a 24/7 working application
• Find any bottlenecks that could cause performance degradation or delays in communication
• Find points of improvements for the code

The secondary goal was to look at the overall design, and if needed propose changes for future updates.

The application

The application to review consisted of:

• Thousands of VIs
• Hundreds of xControls
• Thousands of Shared Variables
• Multiple interacting applications
• Dynamically loaded user interface with 20+ different user panels
• Redundant communication
• Redundant applications.

The Review Workflow

There is no single tool that will find all the potential issues we are looking for, so we used several different tools:

VI Analyzer

• Documentation and most style guide issues can be detected with the VI Analyzer toolkit
• Runs in development with the code in idle mode

Desktop Execution Trace Toolkit (DETT)

• Executes in parallel to the application, and requires the application to run – affects the performance of the application to some degree.
• Memory leaks due to orphaned references and some performance bottlenecks can be detected using DETT.
• DETT can also help detect unhandled errors etc.

Profile Performance and Memory

• Executes in parallel to the application and requires the application to be running – affects the performance of the application to some degree.
• This LabVIEW tool is also used to detect performance and memory issues.

None of the mentioned tools will be able to detect if the code is actually doing what it is supposed to do, so we still have to manually review all the code to find functional issues.

In this application we also used a small internal tool for the review to be able to quickly view and log review comments for VIs.

Preparation

First, we needed to decide how the result should be recorded and presented to the team. With limited time and thousands of VIs there is no time to write a report for each single VI.
Instead we decided what main issues we would like to detect and present, e.g.

• Unclosed references and memory leaks
• Bad coding that requires refactoring
• General bugs

Decide how to record data

All findings have to be recorded so that we can follow up on all issues. In this case all issues were recorded in an Excel document that was transferred to the client’s bug-tracking system.

Prioritize review elements

Next we put different priorities on the different types of issues, e.g.

Prio 1.
The highest priority is used for obvious bugs and memory leaks. Code doesn’t execute as intended, e.g.

• Implementation doesn’t match documentation
• Race conditions
• Bad error handling – errors dropped
• Reference closed in bad locations – causing potential hangs etc.

Prio 2.
Refactoring needed, or minor potential bug, e.g.

• The code is hard to follow to determine functionality and needs cleaning-up
• Perform calculations on arrays instead of scalars

Prio 3.
Coding style, documentation etc. needs work

• Icons too similar to LabVIEW native functions
• Wiring
• Error handling

Prio 4.

• Suggestions for the future, e.g. change of function to do a task

Here there was a decision to focus on Prio 1 and 2, but still record issues of Prio 3 and higher if they were detected.

Analyze the source

One of the first things to do when reviewing a big application like this is to analyze the source code. The analysis should be fairly quick, but still give us a feeling of the code quality and should also help us detect the different parts of the application.

When we view the code to find the different modules, it is important to remember that grouping of code can be done in many different ways, e.g. code can be grouped by ownership of a library (lvlib or lvclass), by name-spacing (e.g. <GROUP>_<METHOD-NAME>) or just by their storage location. In this specific case, the code was only separated by the disk location, and a number of shared resources (e.g. globals) prevented easy scaling of the application

Find Dependencies

When we have a feeling for the code regarding size, quality, modularity etc., it is time to find the dependencies between code modules.

The dependencies will help us prioritize the work, especially if the amount of time is limited. By focusing on code modules that are used most extensively and that contains critical components, we can identify risks efficiently.

The challenge in this case was that code modules were defined by their file locations, and that some dependencies created links back and forth between code modules.

The Review

To speed up the review process of all LabVIEW files we created a simple LabVIEW application that allowed us to open the next VI to review, and add the comments and issues to the file directly.

Figure 1. The review tool

This tool helped us keep track of how many files we reviewed, and we never had to search for a VI in memory or on disk since the tool opened the VIs directly. It also saved a lot of time because it had support to copy comments from other reviewed VIs, and the same types of mistakes are often seen more than once.

It also saved a lot of time because it had support to copy comments from other reviewed VIs, and the same types of mistakes are often seen more than once.

Example of review comments: