Common Problems Found in EVMS Reviews and Recommended Corrective Actions – Part 2

Part 2:  Poor use of Percent Complete; Data Integrity Issues; Poor Scope Language

This is the second part of a five part series regarding common findings discovered in EVMS reviews and the recommended corrective actions to mitigate those findings.  In the previous article, EAC alignment issues, poor variance analysis, and lack of subcontract management were discussed.  This second part focuses on: poor use of the percent complete earned value technique, data integrity issues, and poor scope language.

The topics anticipated for parts three through five are:

Part 3:  IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse.

Part 4:  Misalignment between BCWP and ACWP; Freeze Period Violations; Failed Data Traces.

Part 5:  Inappropriate use of PERT and LOE; Misuse of Management Reserve; Administrative CAMs.

Here are the three topics for this installment;

1. Poor Use of the Percent Complete Earned Value Technique

Earned Value techniques (EVT) fall into three distinct classifications:

• Discrete
• Apportioned
• Level of Effort (LOE)

Within the Discrete technique there are methods ranging from objective to subjective.  The most objective methods are those that employ a “yes/no” question as the primary measure of accomplishment, e.g., “Has the activity begun?” or “Was the hardware delivered?”.  In these cases, there is no subjectivity in the evaluation of performance.

The percent complete technique can introduce very subjective criteria as the source of measuring accomplishment.  These are sometimes called “Manager’s Estimate”, or, derisively, “The Percent Guess” technique.  The problem with this application of percent complete is that visibility into accurate work package performance may be hidden until it is too late to take effective managerial action.  Worse, managers responsible for activities linked downstream from the percent complete work package may not discover the true status of a critical predecessor until it is too late to implement effective mitigation actions.   While a key mantra for effective project management is “no surprises”, misuse of the percent complete earned value technique can introduce the possibility for a great deal of surprise.

Most Common Corrective Action Plans

The goal in selecting the most appropriate EVT is to use the most objective rationale possible given the nature of the work.  There are key strategies that should be employed to reduce the subjectivity of the percent complete technique.

1. Quantifiable Backup Data (QBD):  This is also sometimes referred to as “Predetermined Rationale”, and essentially requires the establishment of lower level milestones, activities, or steps to determine the percent complete.  These steps are established and weighted prior to beginning the effort. The percent complete is determined when the steps for the percent complete work package are accomplished; i.e. the weight assigned to the steps is earned.  It is recommended that a work package has at least one objective measurement point in each earned value period (generally monthly).
2. Shorter Duration Activities:  Because one of the dangers of using the percent complete technique is a potential delay in reporting accurate status until it is too late to take management action, it is recommended that these types of work packages be relatively short in duration.  The DCMA publication “EVMS Program Analysis Pamphlet”(DCMA-EA PAM 200.1, July 2012) defines any activity more than 44 working days to be a “high duration”activity.  The choice of 44 days generally means that an activity with a duration of that or less will start and finish within 2 months.  Included in the pamphlet is the recommendation that the IMS contain no more than 5% of activities greater than 44 days.
3. Percent Complete Threshold:  One method often employed when using percent complete is the introduction of a percentage ceiling that limits the reported accomplishment until the effort is 100% complete.  This limits the “99% Complete”scenario that can introduce false status and create the often incorrect impression that completion is imminent.  A common threshold in this mandate is to limit the percent complete to 75% or 80% until the activity is 100% complete. Some even limit the work package budget; example, maximum of 300 hours and the duration not to exceed three months.

2)  Data Integrity Issues

To standardize its reviewing organizations and the reduction in resources available to conduct on-site reviews, the DCMA is transitioning to establishing predefined success criteria and testing methods to evaluate the compliance of each guideline using data submitted by the contractor.  In addition, as mentioned above, the DCMA has released to its staff a “Program Analysis Pamphlet” that contains 17 measures of data integrity for contractor supplied data.  A few of these are:

Most Common Corrective Action Plans

In many cases, there may be scenarios that result in a temporary data condition that trips one of the above thresholds; such as Level of Effort (LOE) but these should be the exception rather than the rule.

Because of the growing interest in data integrity, it is important to screen for these conditions prior to submitting the data to the customer.  If possible, an organization should allow time in its monthly business rhythm to correct the easy-to-fix errors prior to submittal of the reports.  Those data flags that are legitimate or cannot be corrected prior to submittal should be reported to the customer in Format 5 of the Integrated Program Management Report (IPMR) or Contract Performance Report (CPR).  Some issues can be more systemic and require a longer correction process. These should also be identified to the customer with a corrective action plan which includes a date for when the corrections will be in effect.

And finally the prime contractor must review all reports delivered by the subcontractor for compliance with the appropriate DID Instructions, and be prepared to take action when the reports are noncompliant with the CDRL and/or DID Instructions.  This action can take the form of a contracting officer’s letter requiring adjustments in future reports, rejection and resubmittal of non-compliant reports, or rejection and contractual remedies; such as payment withholds or Award Fee impact, as applicable.

3) Poor Scope Language in the WBS Dictionary and/or Work Authorization Documents

The original scope for the project/contract is defined in the Contract Statement of Work (CSOW) and then organized within the Work Breakdown Structure (WBS).  The WBS Dictionary defines the technical boundaries of each WBS element and identifies the work products.  At a more detailed level, the scope is also identified in the control account plan and the work authorization documents.  There is an expectation that the scope language will become more specific and descriptive as it migrates from the project’s top level, through the WBS, down to the control accounts.

In EVM Surveillance Reviews, as well as Integrated Baseline Reviews (IBR)s, a normal set of activities includes data traces.  The project’s scope, schedule, and budget are traced across all the pertinent documents and reports to ensure that the EVMS is operating as a single, integrated system.  Included in the scope trace is a review of lower level (WBS element or control account) scope statements to ensure they are consistent with the CSOW.  It is also expected that the WBS Dictionary will be maintained and current with all external and internal changes that add scope to WBS elements or control accounts.  Often an organization begins a project with a well-documented set of scope statements, only to let them languish in the face of constant and voluminous changes.

Most Common Corrective Action Plans

Like Variance Analysis and Reporting, having accurate, current, and descriptive scope statements in all pertinent documents requires a disciplined approach and persistent review.  The first step is establishing a procedure that makes this an important part of the earned value process, including the importance of maintaining the currency of the WBS Dictionary by technically qualified personnel.  The quality of these documents is generally the responsibility of the CAMs in concert with the system engineering organization or equivalent; however, the Project Controls staff plays an important role in assisting the CAMs and reviewing the products.

One of the most common findings in this area is the failure to maintain the WBS Dictionary.  This document must reflect the current scope and incorporate all approved changes.  The contractor should establish a process in the monthly business rhythm that “sweeps” all known changes, generally through the work authorization documents, and evaluates those for incorporation into the Dictionary.

Please contact Humphreys & Associates if you have any questions on this article.