Common Problems Found in EVMS and Recommended Corrective Actions – Part 5

By /

This is the last of a five part series regarding common findings discovered in contractors’ Earned Value Management Systems (EVMS), and the recommended corrective actions to mitigate those findings.

The previous articles discussed: 

Part 1 

Part 2 

Part 3

Part 4

Common Errors and CA part 4

Part 5 of this series includes:  Inappropriate use of PERT and LOE; Misuse of Management Reserve; Administrative Control Account Managers.

1)  Inappropriate use of PERT and LOE

The Program Evaluation and Review Technique (PERT) earned value method is a simple method for calculating the BCWP, where:  BCWP = (ACWP/EAC) X BAC.  In this method, the earned value is completely contingent upon cumulative expenditures (ACWP) divided by an estimate of total expenditures.  Because the results of this formula often have little to do with actual progress, its use is limited to non-critical work, and generally is applied only to high volume, low dollar fixed price material.  The PERT method should never be used for any critical path task, labor, or high dollar value material.  Guideline 7 of the EIA-748-C Standard requires that an EVMS “Identify physical products, milestones, technical performance goals, or other indicators that will be used to measure performance”. The primary condition that must be satisfied in a review of earned value techniques is the application of “meaningful indicators” for use in measuring the status of cost and schedule performance.

Level of effort (LOE) tasks consist of management or sustaining type activities that have no identifiable end products or an established relationship to other measurable effort.  The standard for the control of LOE is documented in Guideline 12, which requires that “Only that effort which is immeasurable or for which measurement is impractical may be classified as level of effort”.  There is no standard threshold for a contract or WBS level that would signify “too much” LOE. However, a common practice during review discussions with the control account managers is to challenge any LOE to assess its appropriateness.  There is always pressure on a contractor to minimize the LOE as the nature of LOE can easily mask or distort the performance of discrete work.

Most Common Corrective Action Plans

The most common response to findings regarding both PERT and LOE is to establish a screening/approval process, with thresholds, during the budgeting process.  For PERT, most Earned Value Management System Description Documents (EVM SDD) will specify the limited use of the technique for high volume, low dollar fixed price material.  Many also take the next step and create a threshold for what is considered “low dollar” and short duration.  This is dependent on the nature of the work, but it is not unusual for an SDD to require that any material extended value (quantity of parts times budgeted unit value) or part number greater than $10,000 (or some other threshold) must be tracked discretely in the IMS and may not use the PERT method.  Some also establish a a duration threshold like no greater than 3 months when there are many parts of low value. One of the signs that PERT is being used inappropriately is when variance analysis included in the Integrated Program Management Report (IPMR) or Contract Performance Report (CPR) Format 5 consistently refers to PERT accounts as drivers, or a schedule variance explanation that refers to material not being tracked in the IMS.

Level of Effort should be justified on a case-by-case basis.  A common strategy for setting the appropriate level of LOE is to require the Program Manager’s approval on all LOE accounts.  While control accounts may contain a mixture of LOE and Discrete Effort, many organizations establish rules concerning the maximum allowable LOE in a control account to prevent the distortion of status; often a threshold of 20% is established.  Above that threshold, a separate control account would be required for LOE work.  While it is a goal to have no more LOE than is required, care must be taken not to measure work that is truly LOE.  CAMs have been known to say that they were required to establish a discrete account even though the nature of the work is impractical to measure.  This type of discovery by a review team can also result in a finding for use of an inappropriate earned value technique.

2)  Misuse of Management Reserve

Management Reserve (MR) is a portion of the overall contract budget held for management control purposes and unplanned events that are within the scope of the contract.  H&A has often heard in the course of our consulting or training that there are few rules regarding MR, and the restrictions are unclear.  This is not quite accurate.  The Integrated Program Management Report Data Item Description (IPMR DID, DI-MGMT-81861) lists four restrictions on the use of MR:

  • MR shall not be used to offset cost variances.
  • MR shall never be a negative value.
  • If MR includes the contractor and subcontractor amounts together, the breakout shall be discussed in Format 5.
  • Amounts from MR applied to WBS elements during the reporting period shall be listed in Block 6.b of Format 3 and explained in Format 5.
    • Format 5:  Identify the sources and uses of MR changes during the reporting period.  Identify the WBS elements to which MR was applied and the reasons for its application.

The EIA-748-C Standard adds a few more caveats for MR:

  • Held for unexpected growth within the currently authorized work scope, rate changes, risk and opportunity handling, and other program unknowns.
  • May be held at the total program level or distributed and controlled at lower management levels.
  • Held for current and future needs and is not used to offset accumulated overruns or under runs.
  • Is not a contingency that can be eliminated from prices during subsequent negotiations or used to absorb the cost of program changes.
  • Must not be viewed by a customer as a source of funding for added work scope. This is especially important to understand that it is a budget item and that it is not for added work scope.

In addition, the Defense Acquisition University Evaluation Guide (EIA 748 Guideline Attributes & Verification Data Traces) requires that the internal MR Log be reconciled with the IPMR (or CPR).

The specific nature of the above requirements reflects the types of abuse experienced with the MR budgets.  The EVM SDD should establish specific organizational guidelines for the application of MR; however, those rules and the organization’s practices must fall within the bounds established by the EIA-748-C and the appropriate Data Item Description (DID).

The discrepancies found in recent reviews take two primary forms: inappropriate application of MR in the current accounting period and poor reporting and discussion involving MR use in the IPMR/CPR Formats 3 and 5.  With the exception of rate or process changes, all applications of MR must be made in association with additional work scope authorized to control accounts.  Because of the prohibition in the requirements regarding the offset of overruns or underruns, applications in the current period can be far more suspicious than in future periods.  Care must be taken to fully justify the timing and use of MR in terms of additional scope.  All applications of MR must have a full accounting in Formats 3 and 5 of the IPMR (or CPR).

Most Common Corrective Action Plans

Response to these discrepancies is usually a matter of policy, training, and discipline.  The EVM SDD must contain a policy that enforces the rules in the guidance documents mentioned above, as well as document who has the authority for approval of MR application (generally the Program Manager).  Those responsible for authorization of MR must be familiar with the approval policies, and their support staff must ensure there is complete visibility in the customer reporting and reconciliation to the program logs.  One of the more common corrective actions is to structure the IPMR (or CPR) Format 5 so that reporting of MR transactions meets the intent of the guidance cited above.

3)  Administrative Control Account Managers (CAMs)

The role of the CAM can change across organizations, and there is no standard set of criteria that defines the CAM’s duties.  The National Defense Industrial Association (NDIA) Integrated Program Management Division (IPMD) Earned Value Management Systems Intent Guide states that “The control account manager is responsible for ensuring the accomplishment of work in his or her control account and is the focal point for management control”.  The requirements for management control can be defined as having three essential attributes for the role of the CAM: responsibility, authority, and accountability.  These attributes assume ownership of the technical, schedule and cost aspects of the scope authorized to a Control Account Manager (CAM).  It is not always the case that the CAM must be the technical expert over the scope of the control account; sometimes that is the role of the “performing organization” rather than the “responsible organization”.

Primarily through discussion with the CAMs, it is easy to assess if they fail in performing any or all three of the above essential attributes.  Because being a CAM brings a set of duties that are over and above those of a technical manager or engineer, the CAM’s role is often not a welcomed addition and some organizations hand it over to employees who have little knowledge of, or responsibility for, the effort.  A comment on a Corrective Action Request (CAR) for an organization that was employing Administrative CAMs was “CAM does not stand for ‘Control Account Monitor’”.  The role of the CAM does include necessary administrative responsibilities; such as, reporting status, maintaining records, and developing analysis for the control accounts.  However, these cannot be the only functions the CAM performs.

A CAM should be active in the development of the control account plans in the IMS and EVM Systems, including being the primary architect for defining the tasks, logic for the schedule and the adequacy of the budget.  The CAM should be the primary contact for the Program Manager regarding the control account, including risk management and corrective action planning.  The CAM should also have the authority to assign and coordinate work performed by other organizations.  The CAM should have enough knowledge of the scope and the executing environment to develop a realistic forecast of costs beyond that of mathematical extrapolation.  If control accounts contain subcontracted work, the CAM is also responsible for management of that subcontractor effort.

Most Common Corrective Action Plans

Choosing the right personnel to fulfill the requirements of the CAM role can be difficult.  One of the first considerations is the appropriateness of the organization that is given the responsibility for management.  An example is for material control accounts where the organization responsible during a program’s development phase may not be appropriate for the production phase.  It is very important for the contractor when submitting the corrective action plan (CAP) to treat the job of the CAM as being critical to project success, and not one relegated to people in the organization who do not have responsibility, authority, or accountability.

The Correction Action Plan should also include a list of essential CAM attributes, and be very clear on the responsibilities and authority of the CAM role.  Companies should make a commitment to ensure that the position is considered critical and not just created to fulfill the requirements of earned value.  This can be demonstrated by not only choosing the right individuals to perform the functions, but also providing the necessary resources, training, and support to function successfully.

This completes our 5 part series. Thank you for your readership.

If you have any questions or would like to inquire about our services, please feel free to contact us. 

Common Problems Found in EVMS and Recommended Corrective Actions – Part 5 Read Post »

Common Problems Found in EVM Systems and Recommended Corrective Actions – Part 4

By / /
HA_Blog-Common-Problems-part-4-Twitter

This is the fourth part of a five part series regarding common problems found in EVM Systems and the recommended corrective actions to help mitigate those findings.  The previous three articles discussed:

common problems found in evm systems - part 4

The topics anticipated for part five are: Inappropriate use of PERT and LOE; Misuse of Management Reserve: Administrative CAMs.

1)  Misalignment between BCWP and ACWP

The Earned Value Management System Description Document (EVM SDD) should include a statement that requires Actual Cost of Work Performed (ACWP) to be reported within the same accounting period as Budgeted Cost for Work Performed (BCWP) is earned; which is most applicable for material.  Both ACWP and BCWP contain the term “Work Performed”.  The ACWP is not a measure of how much has been spent but rather reflects how much it cost to accomplish the scope of work reflected in the BCWP.

Accounting systems generally record actual costs for material when invoices are paid; this may or may not align with when earned value is claimed for that material.  If material earned value is claimed at point of usage, it may be necessary to collect actual costs in a holding account and then delay recording ACWP in the earned value system until the material is used.

When material earned value is taken at the point of receipt, invoice payments may be delayed for 45 days (or more). The actual costs associated with this material will be recorded in the accounting system after the earned value credit is taken.  In this case, recording ACWP in the earned value system must be accelerated.  The process of delaying or accelerating the recording of ACWP in the earned value system is often called using “Estimated Actuals” or, more appropriately, “Estimated ACWP”.

There are two obvious examples of this process being done incorrectly.  The first is in the data where BCWP is claimed without corresponding ACWP in the current period, or vice versa.  This may be below the threshold level for variance explanation and is often attributable to Level of Effort (LOE) control accounts, but it creates a situation that attentive customers will need to understand.  The second example is more direct, and occurs when contractors simply explain the situation in Variance Analysis Reports that are subsequently summarized in the Contract Performance Report (CPR) or Integrated Program Management Report (IPMR) Format 5.  The Control Account Manager (CAM) will use words such as “billing lag,” “accrual delay,” or “late invoicing” in the explanation of a cost variance.  Consequently, any time that financial billing terms are used to explain a cost variance, it raises a flag regarding a potential misalignment between BCWP and ACWP.

One issue with ACWP and BCWP misalignment is that it invalidates the use of the earned value data for predictive purposes.  Unless both data elements are recorded within the same accounting period, using indices such as the CPI, TCPI, or IEAC  (Independent Estimate at Completion) will deliver erroneous results.  The time and effort of the CAMs in the variance analysis process should be spent on managing the physical progress and efficiencies of the work, not having to explain payment or accounting system irregularities.

Most Common Corrective Action Plans

When this issue is reported, the best response is to develop a disciplined Estimated ACWP process, including logs and a monthly trace from the Accounting General Ledger to the EVM ACWP.  It is also important to train the CAMs and support staff on how to record and subsequently retire those entries in an Estimated ACWP log book.  Reviewers of the Variance Analysis Reports should be trained to screen for entries that indicate an inappropriate alignment between BCWP and ACWP.  In addition, as indicated in the blog discussion on Data Integrity (Part 2 of this series), situations where there is BCWP without corresponding ACWP, or vice versa, at the control account level, should be flagged and justified by the CAM prior to submittal of the CPR/IPMR to the customer.

2)  Freeze Period Violations

“Freeze Period” refers to future accounting periods, including the current accounting period, in which baseline changes should be strictly controlled.  This is also sometimes called the “Change Control Period”.  The definition of this period should be in the company’s EVM SDD, but will usually have a time-frame such as “current accounting period plus the next accounting period”.  The SDD should specify what kinds of changes are allowed within this period, how they are to be documented in the CPR/IPMR, and any necessary customer notification or approval requirements when these changes are incorporated.  The SDD should require that customer approval is necessary for changes to open work packages that affect BCWS or BCWP in the current or prior accounting periods, and any changes to LOE data in prior periods or in the current period if the LOE account has incurred charges (ACWP).

There is an additional requirement specific to retroactive adjustments which includes the current period.  The EIA-748-C Guideline 30 specifically stipulates the requirement that these types of changes be controlled, and that adjustments should be made only for “correction of errors, routine account adjustments, effects of customer or management directed changes, or to improve the baseline integrity and accuracy of performance measurement data”.  Again, the reasons allowed for the changes should be specified in the EVM SDD.  However, regardless of the reason, it is a requirement that all retroactive changes be reflected in the current period data in the CPR/IPMR Formats 1 and 3, and that Format 5 include the related explanations (National Defense Industrial Association (NDIA), Integrated Program Management Division (IPMD), Earned Value Management Systems Intent Guide, August 2012).

Some projects have a great deal of volatility.  The incorporation of subcontractor data (especially if that data lags the prime contractor reporting period) and accounting system adjustments often create retroactive (including current period) adjustments.  The operation of change boards may also result in changes, both internal and external, which require immediate implementation.  EVM compliance in this environment is a matter of disciplined incorporation of changes, including visibility and communication to the customer (and sometimes prior approval) of any impacts to the baseline.

Most Common Corrective Action Plans

When discrepancies are found with freeze period noncompliances, the first action should be to ensure that procedures are in place that are compliant with the EIA-748.  The discipline required by these procedures must be communicated to the program team so that a consistent change control processes is maintained.  Key to compliance is visibility and communication of freeze period changes via CPR/IPMR Formats 3 and 5.

H&A has seen a loose interpretation of the guideline allowance for adjustments to “improve the baseline integrity and accuracy of performance measurement data”.  Care must be taken that adjustments falling under this category are not made to avoid variances.

3)  Failed Data Traces

The reviews associated with EVM surveillance and compliance have become increasingly data centric for the past several years.  One of the first steps in a review is submittal to the customer of a complete set of EVM data so analysis can be conducted against predefined success criteria prior to conducting an on-site review.  When there is an on-site review, the data trace portion of that review can be a major component at the company, project, and Control Account Manager levels.

The primary purpose of the data traces is to evaluate the Earned Value Management System.  Is the EVMS operating as a single integrated system that can be counted on for reliable and valid information?  The data traces performed generally follow three separate threads: Scope, Schedule, and Budget.  There are a variety of documents and reports that contain this information, but the reviewers will look for a single thread of data to flow and be traceable throughout the system.

All systems are different, but a common strategy for data traces might be as follows:

  • Scope:  WAD → WBS Dictionary → Contract Statement of Work.
  • Schedule:  WAD → IMS → CAP.
  • Cost (Budget):  RAM → WAD → IMS → CAP → CPR/IPMR Format 1 → CPR/IPMR Format 5.
  • Cost (ACWP):  CAP → Internal Reports → CPR/IPMR (Formats 1 & 2) → General Ledger.

If there are also supplemental sources of data that flow into the EVMS, such as subcontractor, manufacturing, or engineering reports, then these should also be a part of the data trace.

The key to this process is the concept of “traceability”.  The easiest path to prove traceability is if the data are an exact match; however, this is not always possible.  Prime contractors often have to make adjustments to subcontractor data, use of estimated ACWP often will not allow a match with the accounting ledger, and supplemental schedules often “support” the IMS while not matching exactly.  These are normal and explainable disconnects in the data.  When submitting data for review, it is important to know where the data does not match and to pass that information on to the reviewers.  If preparing for an on-site review, the CAMs and others who may be scheduled for discussions should perform a thorough scrub of the data and have quick explanations available when a trace is not evident in that data.

Most Common Corrective Action Plans

It is important that any special circumstances that cause traceability issues be relayed to the review team with the data submittal.  The people who conduct the analysis often operate independently until they are on-site for the review, and it is possible to avoid misunderstandings by identifying any issues with the submitted data set.  This type of communication has the potential to eliminate unnecessary findings.

A short term response to a data trace issue is to establish a process to screen the EVM data before submission to the customer.  Starting with the accounting month end, the statusing and close-out process requires a comparative analysis of the various databases containing the same information.  Because of the volume of data contained in most systems, this should be automated.  There should be time in the monthly business rhythm to allow for corrections and data reloads to improve the accuracy across the various data locations.

The best approach to improved data traces is to design a system that minimizes the number of entries for a single set of data.  For example, H&A found one contractor with over 10 different databases where the CAM’s name was hand entered which resulted in a configuration control nightmare for that data element.  The process of system design should include a complete listing of common data elements that are included in the storyboarding of the process flow.

The topics anticipated for Part 5 are: Inappropriate use of PERT and LOE; Misuse of Management Reserve: Administrative CAMs.

To read previous installments:

  • Part 1 – EAC Alignment Issues, Poor Variance Analysis, Lack of Effective Subcontract Management
  • Part 2 – Poor use of Percent Complete; Data Integrity Issues; Poor Scope Language
  • Part 3 – IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse

Common Problems Found in EVM Systems and Recommended Corrective Actions – Part 4 Read Post »

Common Problems Found in EVM Systems and Recommended Corrective Actions – Part 3

By /

Part 3: IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse

This is the third part of a five part series regarding common findings discovered in Earned Value Management Systems (EVMS) reviews and the recommended corrective actions to help mitigate those findings.  The previous two articles discussed:

Part 3: IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse

The topics anticipated for parts four and five are:

Part 4: Misalignment between BCWP and ACWP; Freeze Period Violations; Failed Data Traces.

Part 5: Inappropriate use of PERT and LOE; Misuse of Management Reserve; Administrative CAMs.

1.  IMS Health Problems 

Several years ago the Defense Contract Management Agency (DCMA) issued the 14-Point Assessment Metrics. Twelve of the metrics are related to the “health” of the Integrated Master Schedule (IMS) while the remaining two (critical path length index and baseline execution index) are “tripwire” metrics for schedule performance.  Of the 12 health metrics, H&A has found that most discrepancy reports (DRs) are associated with missing logic, high total float, and high duration.

Missing Logic:  The DCMA uses logic checks to identify any incomplete tasks that are missing a successor or predecessor, or both.  As a rule of thumb, all activities should be tied to at least one predecessor and one successor with the exception of the first and last activities (respectively) in the project.  By the DCMA’s standards, there is an allowance of 5% for activities not having these types of relationships; but some believe that may be too loose.  In the Planning and Scheduling Excellence Guide or PASEG (National Defense Industrial Association, June 2012, version 2.0) states that all discrete tasks (excluding receipts/deliveries, LOE and summary tasks) should have at least one predecessor and one successor as even one missing logical tie could adversely affect the program’s ability to successfully execute the contract.

High Total Float:  Total float (or “total slack” for Microsoft Project users) is the amount of time an activity can be delayed or expanded before the finish date of the project is affected.  In the DCMA Program Analysis Pamphlet (DCMA-EA PAM 200.1, April 2012), any incomplete tasks with total float greater than 44 working days are considered as having high total float;  an allowance of 5% is also given before this metric trips a red flag.  The primary drivers for inappropriate high total float are missing successor linkages or planning well in advance of need.  For this metric, however, there are many conditions that may drive a high total float value that are perfectly legitimate.  This is especially true for longer projects and production projects that often receive materials well advance of need.

High Duration:  The DCMA Program Analysis Pamphlet classifies incomplete activities with a baseline duration greater than 44 working days as having “high duration”, and again have applied a 5% threshold to the metric.  The fear of very long tasks is that they may not provide enough precision for measurement of accomplishment and will introduce subjectivity into the statusing process.  As with total float, there may be conditions that drive high duration activities that are justifiable.  This is often the case when activities are representative of schedules outside of the IMS, such as at subcontractors or manufacturing planning systems.

Most Common Corrective Action Plans

The IMS is a critical management tool, and the purpose of the health metrics is to ensure that it provides an accurate plan and reliable forecasting for program management and execution.  The basic approach to resolving DRs written for IMS health issues is to first take all the necessary steps to improve the real health of the schedule using the metrics as indicators.  This includes a thorough review of the linkages, relationships, and task durations on an ongoing basis.  Organizations should establish a health check “rhythm”, to be used to review the IMS prior to customer submittal.  This process should also require the CAMs and their scheduling support staff to justify any conditions that may drive tripping a metric.

The contractor should work with its customer to gain a mutual understanding of the conditions that may legitimately result in high total float and high duration activities.  Contractors should try to avoid taking illogical actions, such as adding unnecessary linkages or arbitrarily breaking tasks into small durations simply to meet the metric requirements.  The IMS health metrics are simply indicators of potential issues.  If the nature of the program were one where relatively higher total float values or high durations are to be expected, the appropriate thresholds for tripping a metric may be higher than the standard 44 days.  In these cases, it is worth having a discussion with the customer to establish  new metric thresholds.

2. Data Item Noncompliance

The reports that are generated from an EVMS or IMS, and delivered to the customer, are usually placed on the contract by the incorporation of a Data Item Description (DID) and included in the Contract Data Requirements List (CDRL).  As of June, 2012, both earned value and schedule reporting are included in the Integrated Program Management Report (IPMR), DI-MGMT-81861.  It is important to generate the system reports in accordance with the appropriate DID as the requirements have changed with progressive releases.

Prior to the IPMR DID, IMS reporting was required per DI-MGMT-81650.  For EVM reporting, the previous DID was DI-MGMT-81466A, Contract Performance Report (for contracts established between March, 2005 and June, 2012), and before that was DI-MGMT-81466 or the Cost Performance Report.  The release of the Contract Performance Report DID in March 2005 also eliminated the use of the Cost/Schedule Status Report (C/SSR, DI-MGMT-81467) for new contracts.  There are, however, active contracts which use any one of the above DIDs as the requirements document for earned value and schedule reporting, and compliance of the submitted reports is evaluated against the DID that is required on each contract.

Data Item Descriptions are not just guidelines for reporting, they stipulate the contractual requirements for the documents. There are 203 uses of the word “shall” in the current IPMR DID, and some of these “shall statements” refer to a list of many requirements.  Any planned deviation, or tailoring, from the DID must be approved by the Procuring Authority and documented in the CDRL (DD 1423-1 on DoD contracts).  Section 3.0 of the “IPMR Implementation Guide” (OUSD AT&L PARCA, January 24, 2013) provides tailoring guidance for the IPMR.

Software programs used to generate the IPMR formats have reduced the amount of data specific errors in the reports; however, there are many requirements in the IPMR that are not related to data reporting.  In the requirements for the Format 5 (Explanations and Problem Analyses), for example, there are nine discussion requirements in addition to the required explanations for cost and schedule variances that exceed the variance thresholds.  The narrative portions of the IPMR cannot be generated by a software tool.

A contributing factor in the delivery of poor data items is when the customer encourages noncompliance or does not provide feedback on submitted reports.  It is easy to fall into apathy regarding compliance to the DID when there is no motivation to do so.  This situation, however, does little to convince other reviewers, such as the DCMA, that noncompliance is allowable.

Most Common Corrective Action Plans

When DID noncompliance is found and communicated to the contractor, the best immediate approach is correction and resubmittal of the document.  Noncompliance is most likely a discipline issue which requires a structured approach to developing the report, training the personnel who are responsible, and a thorough review prior to submittal.  Many organizations develop checklists that are used to ensure that all the requirements have been met prior to submittal.  There are also training materials available which can provide cell-by-cell instructions to make the proper entries into these reports (H&A has a DVD titled “Contract Performance and Funds Status Reports (CPR/CFSR) Completion and Reconciliation”).  It may be worthwhile to develop a “buddy system” with another program or another part of the company to exchange outside review and evaluation of data items.  This type of accountability can be mutually beneficial.

3. Planning Package Mismanagement

A planning package is far-term effort in a control account that cannot yet be subdivided into detailed work packages.  Planning packages share similar attributes as work packages, such as a time phased budget, a scope of work, start and finish dates, and must have enough detail in the IMS to support the development of a critical path.  There can be no accomplishment or actual costs recorded against the scope and budget that is defined in a planning package.  When enough information is available to detail plan the planning packages, they are converted to work packages.  This is done through a process called “Rolling Wave Planning”, and it is a good practice to have the detailed information available for at least six months in advance.  Advanced detailed planning is an effective approach to avoid unpleasant surprises, such as lack of availability of the necessary resources or the necessity to begin a hiring exercise.  In addition, near term lack of detail in the Integrated Master Schedule may drive improper or incomplete logic ties, which will impact total float and critical path analyses.

Company EVMS System Description Documents (SDDs) should provide guidance for rolling wave planning, including rules for any baseline adjustment in the current or near term periods.  It is important that planning packages are not allowed to exist in the current or past periods.  It is also improper for any actual costs (ACWP) or performance (BCWP) to be recorded against a planning package.  Most earned value engine software tools prohibit this, but some contractors have been known to override that prohibition in the toolset.  In addition to ACWP and BCWP, there should also be no cumulative BCWS in the current period for any planning package.  Cumulative BCWS is the most noticeable evidence that a planning package was not converted to a work package in a timely manner.

Most Common Corrective Action Plans

The most common corrective action is to conduct a monthly analysis of the EVM data to identify planning packages that are nearing the planning period.  While it is the responsibility of the control account manager (CAMs) to convert planning packages to work packages, Project Controls can easily provide the CAMs with a list of planning packages needing conversion.  If there is no guidance or process written for rolling wave planning, these should be developed to provide instructions to the CAMs and the support staff.  It is also critical that organizations maintain the restrictions in the earned value engines to prohibit the accrual of earned value or actual costs for planning packages.

Please contact Humphreys & Associates if you have any questions on this article.

Common Problems Found in EVM Systems and Recommended Corrective Actions – Part 3 Read Post »

Common Problems Found in EVMS Reviews and Recommended Corrective Actions – Part 2

By /

Part 2:  Poor use of Percent Complete; Data Integrity Issues; Poor Scope Language

This is the second part of a five part series regarding common findings discovered in EVMS reviews and the recommended corrective actions to mitigate those findings.  In the previous article, EAC alignment issues, poor variance analysis, and lack of subcontract management were discussed.  This second part focuses on: poor use of the percent complete earned value technique, data integrity issues, and poor scope language.

The topics anticipated for parts three through five are:

Part 3:  IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse.

Part 4:  Misalignment between BCWP and ACWP; Freeze Period Violations; Failed Data Traces.

Part 5:  Inappropriate use of PERT and LOE; Misuse of Management Reserve; Administrative CAMs.

Here are the three topics for this installment;

1. Poor Use of the Percent Complete Earned Value Technique

Earned Value techniques (EVT) fall into three distinct classifications:

  • Discrete
  • Apportioned
  • Level of Effort (LOE)

Within the Discrete technique there are methods ranging from objective to subjective.  The most objective methods are those that employ a “yes/no” question as the primary measure of accomplishment, e.g., “Has the activity begun?” or “Was the hardware delivered?”.  In these cases, there is no subjectivity in the evaluation of performance.

The percent complete technique can introduce very subjective criteria as the source of measuring accomplishment.  These are sometimes called “Manager’s Estimate”, or, derisively, “The Percent Guess” technique.  The problem with this application of percent complete is that visibility into accurate work package performance may be hidden until it is too late to take effective managerial action.  Worse, managers responsible for activities linked downstream from the percent complete work package may not discover the true status of a critical predecessor until it is too late to implement effective mitigation actions.   While a key mantra for effective project management is “no surprises”, misuse of the percent complete earned value technique can introduce the possibility for a great deal of surprise.

Most Common Corrective Action Plans

The goal in selecting the most appropriate EVT is to use the most objective rationale possible given the nature of the work.  There are key strategies that should be employed to reduce the subjectivity of the percent complete technique.

  1. Quantifiable Backup Data (QBD):  This is also sometimes referred to as “Predetermined Rationale”, and essentially requires the establishment of lower level milestones, activities, or steps to determine the percent complete.  These steps are established and weighted prior to beginning the effort. The percent complete is determined when the steps for the percent complete work package are accomplished; i.e. the weight assigned to the steps is earned.  It is recommended that a work package has at least one objective measurement point in each earned value period (generally monthly).
  2. Shorter Duration Activities:  Because one of the dangers of using the percent complete technique is a potential delay in reporting accurate status until it is too late to take management action, it is recommended that these types of work packages be relatively short in duration.  The DCMA publication “EVMS Program Analysis Pamphlet”(DCMA-EA PAM 200.1, July 2012) defines any activity more than 44 working days to be a “high duration”activity.  The choice of 44 days generally means that an activity with a duration of that or less will start and finish within 2 months.  Included in the pamphlet is the recommendation that the IMS contain no more than 5% of activities greater than 44 days.
  3. Percent Complete Threshold:  One method often employed when using percent complete is the introduction of a percentage ceiling that limits the reported accomplishment until the effort is 100% complete.  This limits the “99% Complete”scenario that can introduce false status and create the often incorrect impression that completion is imminent.  A common threshold in this mandate is to limit the percent complete to 75% or 80% until the activity is 100% complete. Some even limit the work package budget; example, maximum of 300 hours and the duration not to exceed three months.

2)  Data Integrity Issues

To standardize its reviewing organizations and the reduction in resources available to conduct on-site reviews, the DCMA is transitioning to establishing predefined success criteria and testing methods to evaluate the compliance of each guideline using data submitted by the contractor.  In addition, as mentioned above, the DCMA has released to its staff a “Program Analysis Pamphlet” that contains 17 measures of data integrity for contractor supplied data.  A few of these are:

Data Integrity Issues by Humphreys & Associates

Most Common Corrective Action Plans

In many cases, there may be scenarios that result in a temporary data condition that trips one of the above thresholds; such as Level of Effort (LOE) but these should be the exception rather than the rule.

Because of the growing interest in data integrity, it is important to screen for these conditions prior to submitting the data to the customer.  If possible, an organization should allow time in its monthly business rhythm to correct the easy-to-fix errors prior to submittal of the reports.  Those data flags that are legitimate or cannot be corrected prior to submittal should be reported to the customer in Format 5 of the Integrated Program Management Report (IPMR) or Contract Performance Report (CPR).  Some issues can be more systemic and require a longer correction process. These should also be identified to the customer with a corrective action plan which includes a date for when the corrections will be in effect.

And finally the prime contractor must review all reports delivered by the subcontractor for compliance with the appropriate DID Instructions, and be prepared to take action when the reports are noncompliant with the CDRL and/or DID Instructions.  This action can take the form of a contracting officer’s letter requiring adjustments in future reports, rejection and resubmittal of non-compliant reports, or rejection and contractual remedies; such as payment withholds or Award Fee impact, as applicable.

3) Poor Scope Language in the WBS Dictionary and/or Work Authorization Documents

The original scope for the project/contract is defined in the Contract Statement of Work (CSOW) and then organized within the Work Breakdown Structure (WBS).  The WBS Dictionary defines the technical boundaries of each WBS element and identifies the work products.  At a more detailed level, the scope is also identified in the control account plan and the work authorization documents.  There is an expectation that the scope language will become more specific and descriptive as it migrates from the project’s top level, through the WBS, down to the control accounts.

In EVM Surveillance Reviews, as well as Integrated Baseline Reviews (IBR)s, a normal set of activities includes data traces.  The project’s scope, schedule, and budget are traced across all the pertinent documents and reports to ensure that the EVMS is operating as a single, integrated system.  Included in the scope trace is a review of lower level (WBS element or control account) scope statements to ensure they are consistent with the CSOW.  It is also expected that the WBS Dictionary will be maintained and current with all external and internal changes that add scope to WBS elements or control accounts.  Often an organization begins a project with a well-documented set of scope statements, only to let them languish in the face of constant and voluminous changes.

Most Common Corrective Action Plans

Like Variance Analysis and Reporting, having accurate, current, and descriptive scope statements in all pertinent documents requires a disciplined approach and persistent review.  The first step is establishing a procedure that makes this an important part of the earned value process, including the importance of maintaining the currency of the WBS Dictionary by technically qualified personnel.  The quality of these documents is generally the responsibility of the CAMs in concert with the system engineering organization or equivalent; however, the Project Controls staff plays an important role in assisting the CAMs and reviewing the products.

One of the most common findings in this area is the failure to maintain the WBS Dictionary.  This document must reflect the current scope and incorporate all approved changes.  The contractor should establish a process in the monthly business rhythm that “sweeps” all known changes, generally through the work authorization documents, and evaluates those for incorporation into the Dictionary.

Please contact Humphreys & Associates if you have any questions on this article.

Common Problems Found in EVMS Reviews and Recommended Corrective Actions – Part 2 Read Post »

Common Problems Found in EVM Systems – Recommended Corrective Actions – Part 1

By /
HA_Blog-Common Problems part 1

Humphreys & Associates Corrective ActionsHumphreys & Associates (H&A) has the opportunity to work with a broad spectrum of clients who operate their Earned Value Management System (EVMS) under the contractual authority of a variety of customers.  Many clients have a surveillance program conducted by the Defense Contract Management Agency (DCMA), the Department of Defense (DoD) Executive Agent for EVMS which also has a reciprocal agreement with other agencies.  Some agencies, such as the Department of Energy (DoE), have their own surveillance programs.  H&A also works with many companies to help them “tune up” their EVMS as a part of sound business practices.

In the course of conducting and supporting client reviews, H&A has identified several recurring themes that many organizations allow into their EVMS, and because we often support the resolution of these issues through the Corrective Action Planning process, we can also recommend the most common remedies to prevent or correct them.

This will be a 5 part series that will cover three topics in each article.

The anticipated topics for parts two through five (subject to change) are:

            Part 2:  Poor use of Percent Complete; Data Integrity Issues; Poor Scope Language.

            Part 3:  IMS Health Problems; Data Item Non-Compliance; Planning Package Misuse.

            Part 4:  Misalignment between BCWP and ACWP; Freeze Period Violations; Failed Data Traces.

            Part 5:  Inappropriate use of PERT and LOE; Misuse of Management Reserve: Administrative CAMs.

1) The Estimate at Completion (EAC) is systemically out of alignment with cumulative performance with no justification.

The Earned Value Management System Guidelines (EVMSG) define the EAC as the sum of the contract’s cumulative to date Actual Cost of Work Performed (ACWP) plus the company project manager’s best estimate of the time-phased resources (funds) required to complete the remaining authorized work, the Estimate to Complete (ETC).  One of the measures that is used to provide a “sanity check” of the reasonableness of the EAC is to compare the To Complete Performance Index (TCPI) to the Cost Performance Index (CPI).

  • CPI = BCWP / ACWP: An indicator of the cost efficiency at which work is being performed.
  • TCPI = (BAC-BCWP) / (EAC-ACWP): The cost efficiency that would have to be attained in order to achieve the EAC value being used in the formula.

According the DCMA publication “EVMS Program Analysis Pamphlet” (DCMA-EA PAM 200.1, July 2012), a mathematical difference of 0.10 or greater is used as an early warning indication that the contractor’s forecasted completion cost could possibly be unrealistic, stale, or not updated recently.  This is generally measured at all levels of the project, but is often the focus at the control account level.

Having a difference greater than 0.10 does not mean the EAC must be adjusted.  Often the TCPI formula will give unusually high or low values when the account being measured is at the beginning or end of its progress.  There may be a very good reason for the forecasted efficiency of an Estimate to Complete (ETC) to be out-of-line with its historical efficiency.  But often surveillance reveals a large number of accounts that are out-of-line by this measurement, indicating a lack of discipline in an organization’s EAC process.

Most Common Corrective Action Plans:

The first action should be to ensure that the Earned Value Management System stipulates an update to the ETC on at least a monthly basis, and the Control Account Managers (CAMs) are held accountable for their EACs’ realism.  This is often accomplished through training, updating processes, and the development of easy to use tool sets.  The organization could also require the CAM to justify any control account’s ETC that is out of tolerance with an EAC realism check.

Care must be taken to ensure that the EACs are not being set just to avoid tripping this metric.  It is easy to “calculate” an EAC by dividing the BAC by the CPI; however, this does not meet the intent of the guideline requirements.

2) Poor Variance Analysis and Reporting (VAR)

This is probably the most common Corrective Action cited by reviewers performing EVMS surveillance.  Because the variance analysis relies completely on the discipline of the CAMs and their support staffs, the process often atrophies for various reasons. Within the variance analysis report (VAR) process, probably the most troublesome is the identification and tracking of Corrective Actions to their logical completion.  A well written variance analysis will explain the root cause, impact and corrective action associated with a cost (current period, cumulative, at completion) or a schedule (current period, cumulative) variance.  Cost variance explanations should provide a breakdown of the rate versus hours for labor or the price versus usage for material.  Schedule variance explanations should focus on identifying the effort that is not being accomplished per the plan, and forecast the Estimated Completion Date (ECD) for when the schedule variance will go to zero.  Most important in all of this process is the identification and tracking of the Corrective Actions that are being taken to either mitigate the variance or at least ensure that the variance does not worsen.

Reviewers of VARs often find that there is a lack of clear and concise treatment of the problem, and that many writers of VARs simply reiterate the variance as indicated by the data, such as “The cost overrun is because more hours were expended than originally planned”.  Many VARs will point a finger back to the system that generated the data.  But the worst offender is the lack of a corrective action plan that includes mitigations steps that address the issue.

Most Common Corrective Action Plans

Poor variance analysis is primarily a discipline issue with the CAMs and their support staffs. Like most discipline issues, this can be addressed by training, adding structure, and incorporating reviews of the VAR process.  The writing of VARs is one area where annual refresher training may be needed in order to verify that the concepts are still being practiced.  This training should involve the managers in the organizational chain above the CAMs who are responsible for reviewing and approving the variance analysis reports.

It can also be beneficial to add a structured approach to the variance explanation page.  This should include required inputs on Root Cause, Impact, and Corrective Action, and may include a link to a corrective action tracking system that allows the user to create and assign mitigation steps plus provide status of previously identified corrective actions.  Some organizations have also introduced the scoring of VARs using a scoring template as a feedback mechanism for those writing the reports.

3) Lack of Effective Subcontract Management

The Interim DoD Instruction for 5000.02, “Operation of the Defense Acquisition System” (November 25, 2013), reiterates the requirements for earned value application on all cost/incentive contracts greater than $20M including subcontracts.  In the DCMA Instruction 1201, “Corrective Action Process” (September 23, 2013), the importance of subcontract management is stressed with the following:

“Prime contractors have wide latitude as to how they control their supply chain and are ultimately responsible for flow down and execution of contract requirements. When DCMA discovers a noncompliance requiring a Level I or II CAR at a subcontract level, the appropriate CAR (Level I or II) shall be issued directly to the subcontractor with notification to the prime contractor via the prime CMO. The notification to the prime contractor shall be redacted as needed to prevent disclosure of subcontractor proprietary information. In situations where a noncompliance(s) at the subcontract level meets the criteria for a Level III CAR, the CAR shall be issued to the prime contractor.”

Over the past few years, the lack of effective and consistent management of subcontractors with an EVM flow down requirement has been the source of numerous discrepancy reports (DRs), and has been cited as a significant attributor to EVMS disapprovals/de-validations.  As noted in the language above, prime contractors are not only responsible for including the correct clauses in subcontracts but also have responsibility for the execution of the contract requirements.  When a subcontractor does not follow its own system description, is not in compliance with the EIA-748 Guidelines, or submits reports that do not meet the requirements of the contractual Data Item Description (DID), it is the responsibility of the prime to enforce compliance.

The reasons for ineffective subcontract management are many, but in some organizations enforcement of contractual requirements is the domain of a subcontract administrator/buyer who may not have a firm understanding of the EVMS requirements.  Many subcontractors also often resist the surveillance attempts by their prime, or attempt to reject any efforts by their prime to enforce the execution of these types of contractual requirements.

Most Common Corrective Action Plans

The first step in corrective action is a thorough understanding of the contractual requirements and language in the subcontracts.  H&A has found numerous examples of incorrect contractual language, or even language implying that the subcontractor reports are for “information only”, thus diminishing the ability of the prime to enforce compliance.  The contract should include: 1) A requirement for compliance with the EVM system guidelines in EIA-748, preferably by including the appropriate DFARs clauses (DFARs 252.234-7001 and 7002); 2) the appropriate DID, currently DI-MGMT-81861, referenced in the Contract Data Requirements List (CDRL) with any special tailoring or reporting guidance documented (for contracts issued prior to June, 2012, the Contract Performance Report DID is DI-MGMT-81466A and the IMS DID is DI-MGMT-81650); 3) language which establishes the right of the prime contractor to conduct reviews, such as Integrated Baseline Reviews (IBRs) and recurring system surveillance.

The prime contractor must then establish a disciplined surveillance program that includes review of all 32 EIA-748 Guidelines on an annual basis at each subcontractor’s facility.  This process should also include the procedure for issuing and tracking corrective actions.  These surveillance reviews are often conducted jointly with the cognizant DCMA/DCAA for each subcontractor.

And finally the prime must review all reports delivered by the subcontractor for compliance with the appropriate DID, and be prepared to take action when the submitted reports are noncompliant with the CDRL and/or DID Instructions.  This action can take the form of a subcontract administrator’s letter requiring adjustments in future reports, rejection and resubmittal of non-compliant reports, or rejection and contractual remedies; such as payment withholds or Award Fee impact, as applicable.

Please contact Humphreys & Associates if you have any questions on this article.

April 25th, 2014 – Part 2:  Common Problems found in EVMS | Poor use of Percent Complete; Data Integrity Issues; Poor Scope Language.

Common Problems Found in EVM Systems – Recommended Corrective Actions – Part 1 Read Post »

Factoring Subcontractor Data

By /

Factoring Subcontractor Data by Humphrey's & AssociatesIf you are a prime contractor Control Account Manager (CAM), how do you plan Budgeted Cost for Work Scheduled (BCWS) and claim Budgeted Cost for Work Performed (BCWP) when the negotiated subcontract value you are assigned to manage differs from your control account budget?

First, let’s understand the facts. You are assigned to manage a deliverable hardware subsystem reporting element in the contract work breakdown structure that will also be reported in the Integrated Program Management Report (IPMR).  The history of your subcontract is:

  • The subcontractor proposed a total price of $110M (Cost $102M and Fixed Fee $8M).
  • For various reasons, your program manager believed that the final subcontract could be negotiated for 18% less than the proposed price, so your authorized budget is $90M (including fee).
  • You placed the authorized budget in a planning package until the completion of subcontract negotiations.
  • The subcontractor will report its IPMR data at the total price level (including fee).
  • The subcontract was negotiated for a total price of $100M (Cost $95M and Fixed Fee $5M).
  • You requested that the program manager make up the budget difference from Management Reserve (MR), but the program manager declined.

Factoring the subcontractor’s data is the best approach in this situation.  Factoring the subcontractor’s data simply means applying a consistent multiplier to the subcontractor’s budget values: Budgeted Cost for Work Scheduled (BCWS) and Budget at Completion (BAC).  The multiplier is developed by dividing the available control account budget by the subcontract price.  In the example, the multiplier is .9 and was derived by dividing the CAM’s budget of $90M by the total subcontract value for the hardware system of $100M.  This multiplier is applied to the time-phased budget provided by the subcontractor as shown in the table below.  This factored budget becomes the prime contractor’s control account budget.

Factoring Subcontractor Data Budget (Example)

The calculation of the prime contractor’s earned value, also known as the Budgeted Cost for Work Performed (BCWP), simply requires applying the same factor to the subcontractor’s cumulative BCWP each month as shown in the table below.  In the example, for the month of March, cumulative subcontractor BCWP of 32 multiplied by the factor of 0.9 yields the prime contractor’s factored BCWP of 29. When calculating the BCWP, the value in the prime contractor’s Earned Value Management System must reflect the same percent complete (BCWP/BAC x 100) as the subcontractor’s reported data; this is illustrated by the highlighted cells in the data example.

Factoring Subcontractor Data - BCWP

Note that factoring does not apply to Actual Cost of Work Performed (ACWP), the Estimate to Complete (ETC) or the Estimate at Completion (EAC) because those values represent actual costs rather than budgeted amounts that must reconcile with the subcontract Target Cost.

Factoring can occur whether the prime contractor’s budgeted amount for the subcontract effort is either greater or less than the subcontract negotiated price.  Another instance where factoring is appropriate is when the subcontractor provides IPMR data without fee.  The subcontractor’s fee is a cost to the prime contractor and should be included in the prime contractor’s Performance Measurement Baseline (PMB), so factoring is an appropriate technique in this situation.

In summary, remember the following when factoring subcontractor data:

  • Factoring ensures that the subcontract factored BCWS equals the prime contractor’s budget.
  • The prime contractor’s factored BCWP must yield a percent complete consistent with the subcontractor’s percent complete.
  • Factoring does not apply to the ACWP, ETC or the EAC.

Thank you for reading our blog. You can also sign up for our EVMS Newsletter. Give Humphrey’s & Associates a call with questions or to inquire about our classes, certifications and services.

 

Factoring Subcontractor Data Read Post »

Scroll to Top