EAC

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs

, , ,

A previous blog, Maintaining a Credible Estimate at Completion (EAC), discussed why producing a realistic EAC is essential to managing the remaining work on a contract. Internal management and the customer need visibility into the most likely total cost for the contract at completion to ensure it is within the negotiated contract cost and funding limits.

As noted in the earlier blog, one common technique to test the realism of the EAC is to compare the cumulative to date Cost Performance Index (CPI) to the To Complete Performance Index (TCPI).

Example of Using the Metrics for Evaluating Data

One example of documented guidance to industry for evaluating the realism of the EAC is the DOE Office of Project Management (PM) Compliance Assessment Governance (CAG) 2.0, and the related DOE EVMS Metric Specifications they use to assess the quality of schedule and cost data. This blog highlights the use of this guidance and how any contractor can incorporate similar best practices to verify EACs at a given WBS element, control account, or project level are realistic.

To refresh, the CPI is the efficiency at which work has been performed so far for a WBS element, control account, or at the total project level. The formula for the cumulative to date CPI is as follows.

Best practice tip: To ensure a valid CPI calculation, verify the BCWP and ACWP are recorded in the same month for the same work performed.

The TCPI provides the same information, however, it is forward looking. While the CPI is the work efficiency so far, the TCPI is the efficiency required to complete the remaining work to achieve the EAC. The formula for the TCPI is as follows.

TCPI Formula

Best practice tip: To ensure a valid TCPI, verify the BCWP and ACWP are recorded in the same month for the same work performed, and the BAC and EAC are for the same work scope. In other words, the scope of work assumptions are the same for the budget and remaining cost. This is why anticipated changes should not be included in the EAC.

The DOE uses the CPI in two of their assessment metrics and the TCPI in one, however, these are critical metrics partly because they are the only ones used to assess two different data evaluations: 1) commingling level of effort (LOE) and discrete work, and 2) EAC realism.

Commingling LOE and Discrete Work

The first use of CPI (no TCPI in this metric) falls under the Budgeting and Work Authorization subprocess. The primary purpose is to evaluate the effect of commingling LOE and discrete work scope has on control account metrics. The basic premise for this metric is that if the CPI for the LOE scope is significantly different than that for the discrete, the mixture of LOE in that control account is likely skewing overall performance reporting.

Here is the formulation DOE uses.

C.09.01:  Control Account CPI delta between Discrete and LOE >= ±0.1

X = Number of incomplete control accounts (WBS elements) in the EVMS cost tool, where

  1. The LOE portion of the budget is between 15% and 80% of the total budget, and
  2. The difference between the CPI for the discrete work and the LOE work is >= ±0.1.
Y = Number of incomplete control accounts (WBS elements) in the EVMS cost tool.
Threshold = 0%

Best practice tip: Run this metric quarterly on your control accounts that commingle LOE and discrete work packages. When there is a significant discrepancy between the performance of the LOE versus discrete work effort, consider isolating the LOE effort from the discrete effort at the earliest opportunity. An example could be the next rolling wave planning window or as part of an internal replanning action. Alternatively, it may be necessary to perform the calculations at the work package level to assess the performance of just the discrete effort when it is impractical to isolate by other means.

Process and procedure tip: Ensure the LOE work packages within a control account are kept to minimum (typically less than 15%), during the baseline development phase. This helps to prevent discrete work effort performance measurement distortion during the execution phase. A useful best practice H&A earned value consultants have helped contractors to implement during the budget baseline development process is to perform an analysis of the earned value methods used within a control account and the associated work package budgets. This helps to verify any LOE work packages are less than the 15% threshold for the control account. In some instances, it may be logical to segregate the LOE work effort into a separate control account. The objective is to identify and resolve the issue before the performance measurement baseline (PMB) is set.

EAC Realism

One DOE metric uses the TCPI and this involves a comparison to the CPI. This falls in the Analysis and Management Reporting subprocess. This DOE EVMS Metric Specification states: “This metric confirms that estimates of costs at completion are accurate and detailed.” As noted above, the metric compares the cost performance efficiency so far to the cost efficiency needed to achieve the EAC and is specific to the EAC a control account manager (CAM) would review for their scope of work. Depending on the level actual costs are collected, this analysis may need to be performed at the work package level instead of the control account level.  

Here is the formulation DOE uses assuming actual costs are collected at the work package level.

F.05.06:  Work Package CPI – EAC TCPI > ±0.1
X = Number of incomplete (>10% complete) work packages where CPI –TCPI > ±0.1.
Y = Number of incomplete (>10% complete) work packages in the EVMS cost tool.
Threshold = 5%

There is no requirement that the forecast of future costs has a linear relationship with past performance. While there may be legitimate reasons why future cost performance will fluctuate from the past, outside reviewers who receive EVM data will look for a trend or preponderance of data that would indicate the EACs are not realistic. When a significant number of active work packages are outside the ±0.1 CPI-TCPI threshold, it is an indication that the EACs are not being maintained or are driven by factors other than project performance.

Best practice tip: Run this metric every month for each active work package prior to month-end close. For those work packages outside the ±0.1 threshold, review the EAC to ensure it is an intentional forecast of costs given the current conditions.

Process and procedure tip: One of the training courses H&A earned value consultants often conduct is a Variance Analysis Reporting (VAR) workshop. This workshop is designed to help CAMs become more proficient with using the EVM metrics to assess the performance to date for their work effort, identify the root cause of significant variances, and document their findings as well as recommended corrective actions. This analysis includes verifying their estimate to complete (ETC) is a reasonable assessment of what is required to complete the remaining authorized work and their EACs are credible.

 

Additional References

Further discussion on using the CPI and TCPI to assess the EAC realism at the project level can be found in the DOE CAG, Analysis and Management reporting subprocess, Estimates at Completion. This section provides a good overview of comparing the cumulative to date CPI to the TCPI as well as comparing an EAC to calculated independent EACs (IEACs) for further analysis to assess the EAC credibility. 

Interested in learning more about using EVM metrics as a means to verify EACs at the detail or project level are realistic? H&A earned value consultants can help you incorporate best practices into your processes and procedures as well as conduct targeted training to improve your ETC and EAC process. Call us today at (714) 685-1730.

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs Read Post »

Variance Analysis, Corrective Action Plans, Root Cause Analysis

, , , , , , , , ,

Variance Analysis “provides EVMS contract management with early insight into the extent of problems and allows corrective actions to be implemented in time to affect the future course of the program.” [NDIA ANSI EIA 748 Intent Guide] Department of Defense Data Item Descriptions: DI-MGMT-81861, Integrated Program Management Report (IPMR) paragraphs 3.6.10xx; DI-MGMT-81466A, Contract Performance Report, paragraph 2.6.3; and DI-MGMT-81650, Integrated Master Schedule (IMS) — paragraph 2.5 — all require analysis for significant variances including cause, impact and corrective action plans.  By comparing the performance against the plan, it is possible to make mid-course corrections which assist completion of the project on time and within the approved budget. The Variance Analysis Report (VAR) is a “living, working document to communicate cause, impact and corrective action”. [See: Chapter 35 Variance Analysis and Corrective Action, Project Management Using Earned Value, Humphreys & Associates, page 707.] Well-written variance analyses should answer the basic questions of why, what and how.

Cause is also known as root cause, nature of the problem, problem statement, issue, or problem definition. Root cause is the fundamental reason for the problem. Root cause is required in order to take preventative corrective action. The explanation of the variance is broken down into each of its components: discuss schedule variances separately from cost variances; discuss labor separately from non-labor; discuss which portion of the variance was caused by efficiency (hours) and which portion was because of dollars (rates) or if the variance was driven by material discuss how much was because of price and how much was because of usage. For more information refer to Humphreys & Associates blog Variance Analysis-Getting Specific.

Once the root cause of the problem has been identified and described, the impact(s) on the project should be addressed. Identify impacts to customers, technical capability, cost, schedule (including when the schedule variance will become zero), other control accounts, program milestones, subcontractors, and the Estimate at Completion, including rationale.

A corrective action (CA) plan should be developed that describes the specific actions being taken, or to be taken, which includes the individual or organization responsible for the action(s). The corrective actions should be directly derived from root cause analysis and related to each identified root cause.   Results from previous corrective action plans should be included.  Occasionally, a successful plan will include interim modifications or fixes in the short term, with long term changes identified as well. When no corrective action for an overrun is possible, an explanation and EAC rationale should be included.  A corrective action log should be used that tracks the actions taken and the status of the corrective plan for each variance analysis cycle.  As was stated in the Humphreys & Associates article:  Corrective Action Response: Planning and Closure – Part 2 of 2  “It is critical that verification methods, objective measures, metrics, artifacts, and evidential products are identified that will verify that the corrective actions are effective.”  Corrective action plans based on clearly a defined root cause facilitates time management action and avoids the occurrence of repetitive problems.

Variance Analysis, Corrective Action Plans, Root Cause Analysis Read Post »

Scroll to Top