Although adopted by many design departments, test software developers have only recently started to evaluate and implement the standard. To effectively implement CMMI, developers need to understand the standard as well as some of the challenges they might face during implementation and how to overcome them.

What Is CMMI?
In the 1980s, the DoD tasked the Software Engineering Institute (SEI) at Carnegie Mellon University to develop a comprehensive set of software development guidelines based on previous popular development models. With CMMI, the SEI provided tools for companies to standardize the design, implementation, and testing of their software to increase its quality.

Even if an organization does not follow a standardized procedure, certain development processes are in place. These processes include methods for project specification creation, peer code reviews, source code control, and bug tracking. CMMI helps development teams understand these current practices, learn what worked in previous projects, and implement standards that will best increase the quality of their software in the future.

Quality assurance departments tasked with managing CMMI implementation and many high-level managers see test engineering as a hardware department. The importance and quantity of instrumentation in a test department can obscure the amount of software necessary to automate all the instrumentation as well as the software necessary to develop virtual instrumentation.

In addition, because test systems rarely are part of the product delivered to the customer, test software has not been required to follow CMMI development processes. Finally, even though some test engineering departments would like to follow a standard process like CMMI, the time constraints to get a project delivered on time make following a rigorous development process very challenging.

Over the last few years, some test software departments have incorporated CMMI in their development processes. This implementation has been motivated by the increasing importance of software in test and the benefits that design departments have seen in following the standard.

Software continues to increase in importance as test engineers automate processes that required manual intervention in the past. In addition, as test engineers see the benefit of a software-based approach to instrumentation, software has evolved from automating test execution to actually defining instrument functionality. As a result, implementing CMMI for test software development can be challenging.

CMMI Level 1 assumes there is no formalized software development process in place. If a software development process exists at all, it is very ad hoc and not used in more than one project. There is very little control over the development process and a lack of metrics to compare the success of one project to another.

The focus of CMMI Level 2 is to standardize the development process in projects with similar applications. Basic project management process areas such as requirements management, configuration management, and project monitoring and control are implemented at this maturity level.

By standardizing the development process across similar projects, the organization can repeat earlier successes in future projects. CMMI Level 2 is the first level most test software departments implement and contains many of the tactical processes necessary for the rest of the CMMI maturity levels.

CMMI Level 3 goes beyond standardizing the development process in particular projects and seeks to have a common software development standard across the entire organization. Each project either follows the development standard or uses an approved and tailored version of the process for developing and maintaining software.

In CMMI Level 4, the organization uses the standardized development process to define and collect detailed measures of the software and products. For example, each project can measure the rate at which software issues are resolved. By collecting these measures, the organization can quantitatively understand, control, and compare the software development process and products among projects.

Finally, CMMI Level 5 uses the measures developed in Level 4 to implement continuous process improvement based on quantitative data and determines the success of piloting innovative ideas and new technologies. For example, an organization can use the data acquired on the rate of resolution of software issues in Level 4 to statistically compare teams or projects with different tools or new processes.

Test developers not familiar with CMMI have the initial challenge of moving from CMMI Level 1 to Level 2. Whether they use a text-based development environment such as National Instruments LabWindows/CVI, a graphical development environment such as NI LabVIEW, or test management software such as NI TestStand, it is critical that they understand the components of CMMI Level 2 as well as some of the implementation challenges developers might face.

Of these, requirements management and configuration management stand out as two areas that organizations can implement with differing levels of effort depending on the programming language and platform used to develop a test system. In addition, they pose certain challenges that are overcome differently depending on whether the organization is using a text-based or graphical development environment or test management software.

One of the biggest challenges is understanding the relationship between the requirements and the implementation of software, otherwise known as bidirectional traceability. The process of determining bidirectional traceability can be very time-consuming and error prone.

To streamline requirements management and traceability, National Instruments developed NI Requirements Gateway. It extracts information from different formats, such as Telelogic DOORS, IBM Rational RequisitePro, Microsoft Word, ASCII text, and Adobe Acrobat. The program also parses requirement coverage information documented in LabVIEW VIs, indicators and controls, and NI LabWindows/CVI functions as well as NI TestStand workspaces, sequences, and steps (Figure 2).

Figure 2. NI Requirements Gateway Architecture

By using the requirement and coverage information, NI Requirements Gateway shows bidirectional traceability relationships with multiple views so developers can see these relationships interactively. It helps developers determine how requirements are implemented in the test system as well as which ones different components in an application should be implementing. This traceability information can be viewed interactively and documented into a report, such as a traceability matrix.

By streamlining the traceability between requirements and their implementation in software, NI Requirements Gateway facilitates the identification of inconsistencies between project work and requirements. This is one of the goals of the requirements management process area.

NI Requirements Gateway navigates a list of requirements and displays the intended implementation as well as the feature that is implementing it. The developer can directly navigate into the code to determine whether there are any discrepancies between the code and the requirements.

In addition, NI Requirements Gateway helps developers meet a third goal of the process: managing requirement changes. Requirements management software, such as Telelogic DOORS, includes features that manage changes in requirements throughout the development process.

On the other hand, if developers use tools such as Microsoft Word or Adobe Acrobat, there is no out-of-the-box functionality for determining changes. NI Requirements Gateway identifies changes regardless of format by generating snapshots of the requirements document during different stages of the development process and determining the differences between the requirements at various stages.

Even though implementing a configuration management system helps meet the process area goals, interacting and communicating with these configuration management systems can be challenging and time-consuming. Checking in and checking out components can be confusing unless the development environment integrates with configuration management software.

LabVIEW, LabWindows/CVI, and NI TestStand incorporate multiple different configuration management products such as Microsoft Visual SourceSafe, Perforce, Rational ClearCase, and CVS. Developers can easily check in and check out individual files as well as complete projects from the configuration management system directly from the development environment.

In implementing the configuration management process area, developers establish the integrity of software components by comparing them and resolving any conflicts. Comparing differences in code can be challenging, especially if the development environment is not text-based.

To facilitate comparing graphical code written in LabVIEW, the LabVIEW Project helps developers graphically compare a VI against the last version of the VI checked into the source-code control repository. Additions as small as a change in the front panel of a VI or as extensive as adding major functionality are highlighted.

In addition, NI TestStand provides two ways to understand the differences between sequences. With the Sequence File Differ, developers can quickly see the differences between sequences and merge changes from one sequence to another. Finally, LabWindows/CVI, due to its text-based paradigm, uses existing packages for comparing code files and merging changes to establish the software component integrity and implement the configuration management process area.