EVM Terminology

EVM Terms

Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

By / / , , ,
, , ,
Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

“When you need to discuss the schedule, look at the schedule.”

– A Scheduler’s Lament

There are many existing formulas for calculating an Independent Estimate at Complete (IEAC) from earned value data. A recent study of a sample of projects found that the calculated IEACs analyzed at the 25%, 50%, and 75% complete points were not accurate when compared to the final actual cost of work performed (ACWP). The following table lists the thresholds used to assess the accuracy of the IEACs at the different complete points for the sample projects.

Percent CompleteAccuracy Threshold
25%Within +/- 10% of final ACWP
50%Within +/- 7% of final ACWP
75%Within +/- 5% of final ACWP

While working on that study of the accuracy of commonly applied IEAC formulas as well as on a small project as an analyst for a customer, the idea for using data directly from the integrated master schedule (IMS) in conjunction with the cost performance data to create a new IEAC formula emerged.

Using Data Directly from the IMS to Calculate an IEAC

It should be noted that none of the generally used IEAC formulas use data directly from the IMS. The IEAC formulas use data found in the cost performance portion of the earned value monthly reports to customers.

IMS data is only used indirectly in the IEAC formulas. When a task is started and progress updated, the earned value (the budgeted cost for work performed or BCWP) is developed from the progress reported. This is measured against the cost baseline (the budgeted cost for work scheduled or BCWS).

At the same time, in the IMS environment, the schedule analysts are calculating the Baseline Execution Index (BEI) for task completions/finishes. BEI (for finishes) measures how many of the tasks baselined to be completed by the cut-off date were completed. If all the tasks were done (BEI = 1), their value would have been earned. Of course, other tasks could have started, progressed, and maybe even finished. For this example, the Schedule Performance Index (SPI) calculated at that point (BCWP/BCWS) should be at least 1 and potentially higher. The SPI reflects the baseline value of completed tasks plus the in-process claimed baseline value. The in-process claimed value can be subjective in some cases.

The argument, if there were one, might be there is no need to try and include BEI or similar schedule measures in the IEAC formulas since they already include SPI.

However, there is a whole different and unique set of information coming from the IMS that is not currently used in the IEAC formulas. That information is what we chose to call “Duration Performance” and “Realism Ratio.” These are measures of the actual duration for completed tasks and the forecast duration for future tasks.

Calculating Duration Performance

The IMS data includes the baseline number of days assigned to each task as well as the actual number of days to complete each task. If a task is baselined to take 10 days (Baseline Duration = 10) and the task took 15 days to complete (Actual Duration = 15) then it is taking 150% of baseline to do the work.

This is similar to the Cost Performance Index (CPI) that uses the BCWP and the ACWP to determine how efficient the work performance has been. The formula BCWP/ACWP shows how the work accomplished compares to the cost of that work performed.

If we assume, for labor at least, that taking longer to complete a task often leads to costing more than baselined, we can use the Duration Performance to develop an IEAC.

To develop the Duration Performance, we would use the IMS from the month being analyzed to perform the following actions:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with level of effort (LOE) – keep it or ignore it.
  3. Filter for all tasks that are completed (100% complete).
  4. Add up the baseline duration in days for all these completed tasks.
  5. Add up the actual duration days for these same completed tasks.
  6. Compare the actual duration days used to the baseline duration days.

An example would be:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

One of the common IEAC formulas is the “SPI times CPI” that is calculated like this: ACWP + Budgeted Cost of Work Remaining (BCWR) / (CPI x SPI) where BCWR = Budget at Completion (BAC) – cumulative to date BCWP.

Now that we have a duration performance factor, we can develop a new IEAC. The Duration Performance IEAC would be done using the CPI from the same month as the IMS where ACWP + BCWR / (CPI x Duration Performance Index).

Using some actual data from a project for a single month we see:

  • Duration Performance Index = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M

Assessing the Realism Ratio

When we look at the remaining tasks to be completed, we can use the Realism Ratio to assess how the future forecast durations compare to the performance so far.

The data needed are the baseline duration and the forecasted duration for all tasks that have not been started. This concept excludes in-process tasks. In our example from before, the data we created looked like this:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

We would use the same IMS to do this:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with LOE – keep it or ignore it.
  3. Filter for all tasks that are not started.
  4. Add up the baseline duration in days for all these tasks not started.
  5. Add up the forecasted duration days for these same tasks not started.
  6. Compare the forecasted duration days to the baseline duration days.

Let’s say there were 100 tasks not started. If the forecasted days were 1,000 and the baseline days were 1,000 that would yield 100%. When we did the example, the Duration Performance was .67. This means that performance to date was .67 but the future will be 100% or 1. You can see the disconnect. That disconnect we call the Realism Ratio (in this example, .67/1).

Data from the actual project for the same month as discussed earlier shows:

  • Duration Performance = 122% of baseline
  • Future Performance = .86 or 86% of baseline.

This means that the future durations are cut significantly.

We would use this data to develop a factor called a Realism Ratio (86/122 = .70) and that would be used to develop an IEAC using this formula: IEAC (Realism Ratio) = ACWP + BCWR / (CPI x Realism Ratio).

Using the same sample project data from above and adding in an assessment of the forecasted durations for the remaining work, we see:

  • Duration Performance = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • Realism Ratio = .70
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M
  • IEAC (Realism Ratio) = $9.2 +$18.3 / (.92 x .70) = $37.6M

The project is not complete, so the final ACWP position is not known. There is a dramatic difference between the three IEACs. The difference between BEI and SPI indicates that in-process tasks and other factors such as LOE are potentially affecting SPI.

What can we learn from this sample project?

In this example, additional investigation is warranted. There are potential issues with the realism of the baseline and current schedule that are signaling a cost growth issue is likely to occur. Relying on just the time-phased cost data for IEAC calculations may not be sufficient to assess whether a contractor’s range of EACs included in their monthly cost performance reports are realistic. For more discussion, see the blog on Maintaining a Credible Estimate to Completion (EAC) and the blog on Using EVM Performance Metrics for Evaluating EACs.

Are there lurking cost growth surprises in your projects? You may want to consider revisiting your estimate to complete (ETC) and EAC process to verify there is an integrated assessment of the schedule and cost data to identify potential disconnects. H&A earned value consultants can provide an independent assessment of the quality of the data as well processes and procedures to help you verify your EACs are realistic. Call us today at (714) 685-1730.

Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas Read Post »

Formal Reprogramming: OTB or OTS Best Practice Tips

By / / , , , ,
, , , ,
Formal Reprogramming: OTB or OTS Best Practice Tips

As a result of an Earned Value Management System (EVMS) compliance or surveillance review, the Defense Contract Management Agency (DCMA) or DOE Office of Project Management (PM-30) may issue a corrective action request (CAR) to a contractor. H&A earned value consultants frequently assist clients with developing and implementing corrective action plans (CAPs) to quickly resolve EVMS issues with a government customer.

A recent trend our earned value management consultants have observed is an uptick in the number of CARs being issued related to over target baselines (OTB) and/or over target schedules (OTS). On further analysis, a common root cause for the CAR was the contractors lacked approval from the contracting officer to implement the OTB and/or OTS even though they had approval from the government program manager (PM).

So why was a CAR issued?  It boils down to knowing the government agency’s contractual requirements and EVMS compliance requirements.

What is an OTB/OTS and when it is used?

During the life of a contract, significant performance or technical problems may develop that impact schedule and cost performance. The schedule to complete the remaining work may become unachievable. The available budget for the remaining work may become decidedly inadequate for effective control and insufficient to ensure valid performance measurement. When performance measurement against the baseline schedule and/or budgets becomes unrealistic, reprogramming for effective control may require a planned completion date beyond the contract completion date, an OTS condition, and/or a performance measurement baseline (PMB) that exceeds the recognized contract budget base (CBB), an OTB condition.

An OTB or OTS is a formal reprogramming process that requires customer notification and approval. The primary purpose of formal reprogramming is to establish an executable schedule and budget plan for the remaining work. It is limited to situations where it is needed to improve the quality of future schedule and cost performance measurement. Formal reprogramming may be isolated to a small set of WBS elements, or it may be required for a broad scope of work that impacts the majority of WBS elements.

Formal reprogramming should be a rare occurrence on a project and should be the last recourse – all other management corrective actions have already been taken. Typically, an OTB/OTS is only considered when:

  • The contract is at least 35% complete with percent complete defined as the budgeted cost for work performed (BCWP) divided by the budget at completion (BAC);
  • Has more than six months of substantial work to go;
  • Is less than 85 percent complete; and
  • The remaining management reserve (MR) is near or equal to zero.

A significant determining factor before considering to proceed with a formal reprogramming process is the result from conducting a comprehensive estimate at completion (CEAC) where there is an anticipated overrun of at least 15 percent for the remaining work.

When an OTB is approved, the total allocated budget (TAB) exceeds the CBB, this value referred to as the over target budget. Figure 1 illustrates this.

Before Over Target Baseline
Figure 1 – Over Target Baseline Illustration

When an OTS is approved, the same rationale and requirements for an OTB apply. The planned completion date for all remaining contract work is a date beyond the contract completion date. The purpose of the OTS is to continue to measure the schedule and cost performance against a realistic baseline. The process must include a PMB associated with the revised baseline schedule. Once implemented, the OTS facilitates continued performance measurement against a realistic timeline.

Contractual Obligations

An OTB does not change any contractual parameters or supersede contract values and schedules. An OTS does not relieve either party of any contractual obligations concerning schedule deliveries and attendant incentive loss or penalties. An OTB and/or OTS are implemented solely for planning, controlling, and measuring performance on already authorized work.

Should you encounter a situation where it appears your best option is to request an OTB and/or OTS, the DoD and DOE EVMS policy and compliance documents provide the necessary guidance for contractors. It is imperative that you follow agency specific guidance to prevent being issued a CAR or your OTB/OTS request being rejected.

DoD and DOE both clearly state prior customer notification and contracting officer approval is required to implement an OTB and/or OTS. These requirements are summarized the following table.

ReferenceDoD/DCMA1DOE
RegulatoryDFARS 252.234-7002 Earned Value Management System
“(h) When indicated by contract performance, the Contractor shall submit a request for approval to initiate an over-target baseline or over-target schedule to the Contracting Officer.”
Guide 413.3-10B Integrated Project Management Using the EV Management System
6.1.2 Contractual Requirements.
“…if the contractor concludes the PB TPC and CD-4 date no longer represents a realistic plan, and an over-target baseline (OTB) and/or over-target schedule (OTS) action is necessary. Contracting officer approval is required before implementing such restructuring actions…”
Attachment 1, Contractor Requirements Document
“Submit a request for an Over-Target Baseline (OTB) or Over-Target Schedule (OTS) to the Contracting Officer, when indicated by performance.”
EVMS Compliance2Earned Value Management System Interpretation Guide (EVMSIG)3
Guideline 31, Prevent Unauthorized Revisions, Intent of Guideline
“A thorough analysis of program status is necessary before the consideration of the implementation of an OTB or OTS. Requests for establishing an OTB or an OTS must be initiated by the contractor and approved by the customer contracting authority.		EVMS Compliance Review Standard Operating Procedure (ECRSOP), Appendix A, Compliance Assessment Governance (CAG)
Subprocess G. Change Control
G.6 Over Target Baseline/Over Target Schedule Authorization
“An OTB/OTS is performed with prior customer notification and approval.”
See Section G.6 for a complete discussion on the process.
Contractor EVM SD4DCMA Business Process 2  Attachment, EVMS Cross Reference Checklist (CRC), Guideline 31.
“b. Are procedures established for authorization of budget in excess of the Contract Budget Base (CBB) controlled with requests for establishing an OTB or an OTS initiated by the contractor, and approved by the customer contracting authority?”		DOE ESCRSOP Compliance Review Crosswalk (CRC), Subprocess Area and Attribute G.6
“Requests for establishing an OTB or an OTS are initiated by the contractor and approved by the customer contracting authority.”

Notes:

  1. When DoD is the Cognizant Federal Agency (CFA), DCMA is responsible for determining EVMS compliance and performing surveillance. DCMA also performs this function when requested for NASA.
  2. Along with the related Cross Reference Checklist or Compliance Review Crosswalk, these are the governing documents the government agency will use to conduct compliance and surveillance reviews.
  3. For additional guidance, also see the DoD EVM Implementation Guide (EVMIG) , Section 2.5 Other Post-Award Activities, 2.5.2.4 Over Target Baseline (OTB) and Over Target Schedule (OTS). The EVMIG provides more discussion on the process followed including the contractor, government PM, and the contracting authority responsibilities.
  4. Your EVM System Description (SD) should include a discussion on the process used to request an OTB/OTS. The EVM SD content should be mapped to the detailed DCMA EVMS guideline checklist or the DOE Compliance Review Crosswalk (subprocess areas and attributes) line items.

Best Practice Tips

The best way to avoid getting a CAR from a government agency related to any OTB or OTS action is to ensure you have done your homework.

  • Verify your EVM SD, related procedures, and training clearly defines how to handle this situation. These artifacts should align with your government customer’s EVMS policy and regulations as well as compliance review guides, procedures, and checklists. Be sure your EVM SD or procedures include the requirement to notify and gain approval from the government PM and contracting officer, as well as what to do when the customer does not approve the OTB or OTS. Also discuss how to handle approving and managing subcontractor OTB/OTS situations; the prime contractor is responsible for these actions. Your EVMS training should also cover how to handle OTB/OTS situations. Project personnel should be aware of contractual requirements as well as your EVMS requirements and be able to demonstrate they are following them.
  • Maintain open communication with the customer. This includes the government PM as well as the contracting officer and any other parties involved such as subcontractors. Requesting an OTB or OTS should not be a surprise to them. Verify a common agreement has been reached with the government PM and contracting officer that implementing an OTB or OTS is the best option to provide visibility and control for the remaining work effort.
  • Verify you have written authorization from the government PM and the contracting officer before you proceed with implementing an OTB or OTS. You will need this documentation for any government customer EVMS compliance or surveillance review. Your baseline change requests (BCRs) and work authorization documents should provide full traceability for all schedule and budget changes required for the formal reprogramming action.

Does your EVM SD or training materials need a refresh to include sufficient direction for project personnel to determine whether requesting an OTB or OTS makes sense or how to handle OTB/OTS situations? H&A earned value consultants frequently help clients with EVM SD content enhancements as well as creating specific procedures or work instructions to handle unique EVMS situations. We also offer a workshop on how to implement an OTB or OTS .  Call us today at (714) 685-1730 to get started.

Formal Reprogramming: OTB or OTS Best Practice Tips Read Post »

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs

By / /

Contractors new to earned value management (EVM) often give us a call to help them respond to a government customer request for proposal (RFP) that includes the FAR or DFARS Notice of Earned Value Management System (EVMS) contract clause. Depending on the contract value threshold, the contractor will need to implement an EVMS that can at least produce the contract performance data submittals ($20M or greater) or complete a formal EVMS compliance review ($100M or greater) by a cognizant federal agency (CFA) such as the Defense Contract Management Agency (DCMA). Required data deliverables for a DoD or NASA customer reference the Integrated Program Management and Data Analysis Report (IPMDAR) Data Item Description (DID) as the means to submit monthly performance data. 

What is the objective for placing the IPMDAR on contract?

The government customer wants the monthly source schedule and cost data from the contractor for their own contract performance analysis in a standard format. They need to have reliable schedule and cost data for visibility into current contract performance as well as credible schedule and cost projections for the remaining work. Will the contractor complete the remaining work effort within the contractual schedule and cost targets? The government program manager needs this information for their own planning and budgeting as well as forecasting their funding requirements.

A standard format for collecting the data is important for the government customer so they can perform program portfolio analysis. The DoD established their EVM Central Repository (EVM-CR) to routinely collect contractor data submittals for program portfolio analysis. The data is organized using the MIL-STD-881 (the DoD Standard for Work Breakdown Structures or WBS), as a common basis to organize program data. The IMPDAR DID data submission requirements are defined in the File Format Specifications (FFS) and Data Exchange Instructions (DEI). The FFS and DEI specify the required set of data tables using JSON encoding for the IPMDAR cost and schedule data submissions while narrative text is submitted using Microsoft Word or PDF files so the customer can perform text searches. 

The government program manager can tailor the IPMDAR requirements as defined in the DoD IMPDAR Implementation and Tailoring Guide that complements the DID. For example, they can specify the level of data detail (control account or work package level), whether data can be delivered incrementally, variance analysis options, and requirements for the Performance Narrative Report content.

What is included in the IPMDAR deliverable?

There are three components as outlined below

IPMDAR Components

Notes:

  1. At a minimum, the IPMDAR requires data at the control account level with summary element of cost detail. The contract may specify work package level data.
  2. Inputs from a recent schedule risk assessment (SRA) should be included in the native schedule file submission when available. Depending on the schedule tool, the SRA data may need to be a separate file submission (Word or PDF). Results from the SRA along with other schedule analysis discussions (critical path, driving path, and schedule margin) are required to be included in the Detailed Analysis Report narrative. 
  3. The customer may request the results from a schedule data quality assessment and health metrics be included in the Detail Analysis Report narrative. 

What is required to produce the IPMDAR deliverables? 

For contractors new to EVM, one of their first objectives is to figure out what schedule and cost tools they need to be able to provide the required IPMDAR data and narrative analysis to their customer. H&A earned value consultants are sometimes asked to provide recommendations on commercial off the shelf (COTS) tools for this purpose. Much depends on what the contractor already has in place. 

Common schedule COTS tools such as MSP or Oracle Primavera P6 that have already been implemented will require an add-on to produce the SPD. Keep in mind that the IPMDAR does require SRA data and may require results from performing routine schedule data quality assessments. Some COTS add-ons to MSP or P6 are able to produce the typical schedule data quality metrics as well as produce the SPD. Other COTS scheduling tools such as Deltek Open Plan incorporate the SRA functionality, data quality metrics, and the ability to produce the SPD as part of the core product capabilities.

In most instances, contractors new to EVM do have an accounting/financial system in place to at least capture some level of contract or program/project budget and actual cost data. There may also be some capability to produce ETC data required for EAC financial analysis. The issue is organizing the complete set of time phased cost data (budget, earned value, actual cost, ETC) at the control account and work package level by summary element of cost that aligns with the schedule activity data. A contractor may be able to get by with Excel for a small project, however, it is time and cost prohibitive to create Excel macros to produce the CPD. Most COTS EVM cost tools are able to produce the CPD and have successfully completed the DoD EVM-CR data submission validation checks. This is a better alternative to building an in-house tool.

A data analysis tool such as Encore Analytics Empower is also a good option. Empower can import the time phased cost data from Excel or other COTS EVM tools and produce the CPD output. Empower can also import data from common COTS schedule tools. The benefit to using Empower is the ability to analyze the schedule and cost data in one place to verify alignment, produce interactive dashboards and a variety of analysis data views, and produce the IPMDAR Performance Narrative Executive Summary and Detailed Analysis Report.

Top Three Tips for Implementing Tools to Produce the IPMDAR Outputs

Here are a few tips from H&A’s earned value consultants on implementing tools to support the IPMDAR data submittals. Focus on getting the basics right.

  • Continuously verify the quality of the schedule and cost data. Routinely perform schedule data quality assessment and health checks to proactively resolve schedule construction, status, or data issues. Perform routine cost data validation checks such as earned value and no actual costs for a work package or the cumulative to date earned value exceeds the budget at completion (BAC). Correct all data anomalies before producing the deliverables.
  • Continuously verify the schedule and cost data are in alignment. Consistent schedule and cost data coding is critical to ensure integration and traceability. 
  • Anticipate the scope and level of data detail required. This can impact tool configuration, data structures, and data pulled from other business systems such accounting. For example, be prepared to provide the work package level of detail; actual costs will need to be available at this level. Another example is providing schedule risk assessment inputs; this is usually required at intervals specified in the CDRL.

H&A earned value consultants routinely help clients with constructing the schedule to support the IPMDAR data requirements, setting up the process to do routine data quality checks, integrating the schedule and cost data, and verifying the data before producing the performance reporting data submittals. Another common focus is producing clear and concise variance analysis narrative content. 

We can do the same for you. Call us today at (714) 685-1730 to get started.

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs Read Post »

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

By / /
Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

IEAC_Study_Project_Data_Template-Rev-06052023Download

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs

By / / , , ,
, , ,

A previous blog, Maintaining a Credible Estimate at Completion (EAC), discussed why producing a realistic EAC is essential to managing the remaining work on a contract. Internal management and the customer need visibility into the most likely total cost for the contract at completion to ensure it is within the negotiated contract cost and funding limits.

As noted in the earlier blog, one common technique to test the realism of the EAC is to compare the cumulative to date Cost Performance Index (CPI) to the To Complete Performance Index (TCPI).

Example of Using the Metrics for Evaluating Data

One example of documented guidance to industry for evaluating the realism of the EAC is the DOE Office of Project Management (PM) Compliance Assessment Governance (CAG) 2.0, and the related DOE EVMS Metric Specifications they use to assess the quality of schedule and cost data. This blog highlights the use of this guidance and how any contractor can incorporate similar best practices to verify EACs at a given WBS element, control account, or project level are realistic.

To refresh, the CPI is the efficiency at which work has been performed so far for a WBS element, control account, or at the total project level. The formula for the cumulative to date CPI is as follows.

Best practice tip: To ensure a valid CPI calculation, verify the BCWP and ACWP are recorded in the same month for the same work performed.

The TCPI provides the same information, however, it is forward looking. While the CPI is the work efficiency so far, the TCPI is the efficiency required to complete the remaining work to achieve the EAC. The formula for the TCPI is as follows.

TCPI Formula

Best practice tip: To ensure a valid TCPI, verify the BCWP and ACWP are recorded in the same month for the same work performed, and the BAC and EAC are for the same work scope. In other words, the scope of work assumptions are the same for the budget and remaining cost. This is why anticipated changes should not be included in the EAC.

The DOE uses the CPI in two of their assessment metrics and the TCPI in one, however, these are critical metrics partly because they are the only ones used to assess two different data evaluations: 1) commingling level of effort (LOE) and discrete work, and 2) EAC realism.

Commingling LOE and Discrete Work

The first use of CPI (no TCPI in this metric) falls under the Budgeting and Work Authorization subprocess. The primary purpose is to evaluate the effect of commingling LOE and discrete work scope has on control account metrics. The basic premise for this metric is that if the CPI for the LOE scope is significantly different than that for the discrete, the mixture of LOE in that control account is likely skewing overall performance reporting.

Here is the formulation DOE uses.

C.09.01:  Control Account CPI delta between Discrete and LOE >= ±0.1

X = Number of incomplete control accounts (WBS elements) in the EVMS cost tool, where

  1. The LOE portion of the budget is between 15% and 80% of the total budget, and
  2. The difference between the CPI for the discrete work and the LOE work is >= ±0.1.
Y = Number of incomplete control accounts (WBS elements) in the EVMS cost tool.
Threshold = 0%

Best practice tip: Run this metric quarterly on your control accounts that commingle LOE and discrete work packages. When there is a significant discrepancy between the performance of the LOE versus discrete work effort, consider isolating the LOE effort from the discrete effort at the earliest opportunity. An example could be the next rolling wave planning window or as part of an internal replanning action. Alternatively, it may be necessary to perform the calculations at the work package level to assess the performance of just the discrete effort when it is impractical to isolate by other means.

Process and procedure tip: Ensure the LOE work packages within a control account are kept to minimum (typically less than 15%), during the baseline development phase. This helps to prevent discrete work effort performance measurement distortion during the execution phase. A useful best practice H&A earned value consultants have helped contractors to implement during the budget baseline development process is to perform an analysis of the earned value methods used within a control account and the associated work package budgets. This helps to verify any LOE work packages are less than the 15% threshold for the control account. In some instances, it may be logical to segregate the LOE work effort into a separate control account. The objective is to identify and resolve the issue before the performance measurement baseline (PMB) is set.

EAC Realism

One DOE metric uses the TCPI and this involves a comparison to the CPI. This falls in the Analysis and Management Reporting subprocess. This DOE EVMS Metric Specification states: “This metric confirms that estimates of costs at completion are accurate and detailed.” As noted above, the metric compares the cost performance efficiency so far to the cost efficiency needed to achieve the EAC and is specific to the EAC a control account manager (CAM) would review for their scope of work. Depending on the level actual costs are collected, this analysis may need to be performed at the work package level instead of the control account level.  

Here is the formulation DOE uses assuming actual costs are collected at the work package level.

F.05.06:  Work Package CPI – EAC TCPI > ±0.1
X = Number of incomplete (>10% complete) work packages where CPI –TCPI > ±0.1.
Y = Number of incomplete (>10% complete) work packages in the EVMS cost tool.
Threshold = 5%

There is no requirement that the forecast of future costs has a linear relationship with past performance. While there may be legitimate reasons why future cost performance will fluctuate from the past, outside reviewers who receive EVM data will look for a trend or preponderance of data that would indicate the EACs are not realistic. When a significant number of active work packages are outside the ±0.1 CPI-TCPI threshold, it is an indication that the EACs are not being maintained or are driven by factors other than project performance.

Best practice tip: Run this metric every month for each active work package prior to month-end close. For those work packages outside the ±0.1 threshold, review the EAC to ensure it is an intentional forecast of costs given the current conditions.

Process and procedure tip: One of the training courses H&A earned value consultants often conduct is a Variance Analysis Reporting (VAR) workshop. This workshop is designed to help CAMs become more proficient with using the EVM metrics to assess the performance to date for their work effort, identify the root cause of significant variances, and document their findings as well as recommended corrective actions. This analysis includes verifying their estimate to complete (ETC) is a reasonable assessment of what is required to complete the remaining authorized work and their EACs are credible.

 

Additional References

Further discussion on using the CPI and TCPI to assess the EAC realism at the project level can be found in the DOE CAG, Analysis and Management reporting subprocess, Estimates at Completion. This section provides a good overview of comparing the cumulative to date CPI to the TCPI as well as comparing an EAC to calculated independent EACs (IEACs) for further analysis to assess the EAC credibility. 

Interested in learning more about using EVM metrics as a means to verify EACs at the detail or project level are realistic? H&A earned value consultants can help you incorporate best practices into your processes and procedures as well as conduct targeted training to improve your ETC and EAC process. Call us today at (714) 685-1730.

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs Read Post »

Maintaining a Credible Estimate at Completion (EAC)

By / / , , ,
, , ,

Issues with a contractor’s estimate at completion (EAC) process is a common Earned Value Management System (EVMS) surveillance finding H&A earned value consultants are frequently asked to help resolve. The EAC process can become a major issue when the government customer lacks confidence in the contractor’s EAC data.

Why does a credible EAC matter? 

EACs are important because they provide a projection of the cost at contract or project completion, which is also an estimate of total funds required by the customer. It matters because EACs represent real money. When the most likely EAC exceeds the negotiated contract cost, the contractor’s profit margins may be at risk. It also creates a problem for the customer when the most likely EAC exceeds the funding limits. The customer may either need to secure additional funding or modify the work scope. No one likes cost growth surprises.

Figure 1 illustrates comparing the funding limits with the range of contractor’s EACs to verify they are within the bounds of the funding available to complete the scope of work.

Graph Showing Contractor’s Management EACs with Funding Profile
Figure 1: Contractor’s Management EACs with Funding Profile

What determines whether an EAC is credible? 

A credible EAC reflects the cumulative to date actual costs of work performed (ACWP) (costs the contractor has already incurred) plus the current estimate to complete (ETC). The ETC must provide a realistic estimate of what is required to complete the remaining authorized work and represents the time phased estimate of future funds required.

EACs should be based on performance to date, actual costs to date, projections of future performance, risks and opportunities, economic escalation, expected direct and indirect rates, and material commitments. As illustrated in Figure 1, a project manager should routinely evaluate their project’s ACWP, ETC, and EAC along with the funding profile to verify amounts expended and committed are within the parameters of available contract funds. 

What project control practices help to ensure EACs are realistic?

Three recommended best practices H&A earned value consultants either help implement or have observed that ensure the EAC data are credible include:

  1. Actively maintain the bottom up ETC data every reporting cycle. This starts with updating the current schedule resource loaded activities based on performance to date and the latest planning (timing and resource requirements) for work in progress as well as upcoming work effort. This becomes the basis for updating the time phased cost estimate for work in progress that is added to the cumulative to date actual costs or the cost estimate for future work/planning packages. The current schedule and time phased cost estimate should be in alignment. When data is routinely maintained, it minimizes the time required to update it and capture useful information. The control account managers (CAMs) have the basis to substantiate their estimates as well as relevant data they can use to analyze and take action to address a significant variance at completion (VAC).
  2. Actively monitor project EACs from the top down. Project managers that actively maintain a range of data driven EACs (best case, most likely, and worst case) are better prepared to verify the bottom up EACs are realistic, handle realized risks, and prepare for emerging risks. They routinely incorporate metrics such as comparing the Cost Performance Index (CPI) to the To Complete Performance Index (TCPI) to test the realism of the EAC. They can demonstrate their EACs are credible with backup data, rationale, and narratives they provide to management as well as the customer. 
  3. Maintain open communications at all levels of management, subcontractors, and the customer. As a result, project personnel can quickly handle issues or project changes. The project manager is often the main conduit to handle impacts to their project’s EAC such as when corporate management changes direct or indirect rates, changes in resource availability, a spike in commodity prices, or the customer modifies the scope of work or funding.

What are some things to avoid?

H&A earned value consultants often observe practices that negate the purpose and value of maintaining the ETC and EAC data. Issues with the EAC process are often captured in the government customer’s EVMS corrective action requests (CARs). The CARs frequently point out ad-hoc processes or corporate culture issues. Examples:

  1. Management provides a target EAC number the CAMs must match. This approach increases the likelihood the ETC data are unrealistic. There may be a valid reason for this directive as a management what-if exercise. When done as a routine management strategy, it diminishes the value of the ETC data to manage the project’s remaining work and prevent financial surprises. The CAMs should be in a position where they can substantiate their schedule timeline, resource requirements, and cost estimate to complete the remaining work. The project manager should be in a position where they can verify the bottom up ETC/EAC data to establish a level of confidence in their project level EACs they provide to management as well as the customer.
  2. Project personnel take the path of least resistance. This is often a result of a lack of direction or an established process. They either do not create the ETC data or maintain it on a routine basis. A typical approach is to set a cost management tool option where the EAC is static; the CAM may manually update the EAC number once a quarter. The ETC data has limited to no value. This usually surfaces as a major issue when the contractor must provide an Integrated Program Management Report (IPMR)  Format 7 (time phased history and forecast data), or the Integrated Program Management Data and Analysis Report (IPMDAR) Contract Performance Dataset (CPD) to the customer. The customer quickly discovers the ETC data is lacking for their own analysis.
  3. Schedule and cost are created/maintained separately. This often occurs when the schedule and cost tools are not integrated for the duration of the project. A good deal of effort may go into ensuring the schedule and cost data are in alignment to establish the performance measurement baseline (PMB). The integrated master schedule (IMS) resource loaded activities may be used as the basis for the time phased budget baseline in the cost tool. However, the ETC data in the current schedule may not exist or actively maintained. Project personnel only maintain the ETC data in the cost tool and fail to verify it aligns with the current schedule activities (timing) and resource requirements. Once again, personnel are often lacking an established best practice EAC process.

Pay Attention to Your EAC Process

The ETC and EAC data are just as important as the PMB budget plan because it represents real money. As discussed in the blog How Integrated Baseline Reviews (IBRs) Contribute to Project Success, the goal of the IBR is to verify an executable PMB has been established for the entire contractual scope of work. Similarly, the goal of maintaining a credible ETC and EAC is to verify an executable plan is being regularly updated to accomplish the remaining scope of work within the contract’s schedule, cost, and funding targets. The customer must have confidence in the contractor’s ability to deliver and meet the remaining contract objectives.

The best way to avoid an EAC process CAR is to ensure you have an established process personnel follow, and they know how to use the schedule and cost tools to consistently maintain quality ETC and EAC data. H&A earned value consultants have worked with numerous clients to design or enhance their EAC process. H&A also offers EVMS training workshops that include content on how to develop a realistic EAC. Regular EVMS training always helps to reinforce best practices. Call us today at (714) 685-1730 to get started.

Maintaining a Credible Estimate at Completion (EAC) Read Post »

Scroll to Top