EVM Terminology

EVM Terms

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs

By / /

Contractors new to earned value management (EVM) often give us a call to help them respond to a government customer request for proposal (RFP) that includes the FAR or DFARS Notice of Earned Value Management System (EVMS) contract clause. Depending on the contract value threshold, the contractor will need to implement an EVMS that can at least produce the contract performance data submittals ($20M or greater) or complete a formal EVMS compliance review ($100M or greater) by a cognizant federal agency (CFA) such as the Defense Contract Management Agency (DCMA). Required data deliverables for a DoD or NASA customer reference the Integrated Program Management and Data Analysis Report (IPMDAR) Data Item Description (DID) as the means to submit monthly performance data. 

What is the objective for placing the IPMDAR on contract?

The government customer wants the monthly source schedule and cost data from the contractor for their own contract performance analysis in a standard format. They need to have reliable schedule and cost data for visibility into current contract performance as well as credible schedule and cost projections for the remaining work. Will the contractor complete the remaining work effort within the contractual schedule and cost targets? The government program manager needs this information for their own planning and budgeting as well as forecasting their funding requirements.

A standard format for collecting the data is important for the government customer so they can perform program portfolio analysis. The DoD established their EVM Central Repository (EVM-CR) to routinely collect contractor data submittals for program portfolio analysis. The data is organized using the MIL-STD-881 (the DoD Standard for Work Breakdown Structures or WBS), as a common basis to organize program data. The IMPDAR DID data submission requirements are defined in the File Format Specifications (FFS) and Data Exchange Instructions (DEI). The FFS and DEI specify the required set of data tables using JSON encoding for the IPMDAR cost and schedule data submissions while narrative text is submitted using Microsoft Word or PDF files so the customer can perform text searches. 

The government program manager can tailor the IPMDAR requirements as defined in the DoD IMPDAR Implementation and Tailoring Guide that complements the DID. For example, they can specify the level of data detail (control account or work package level), whether data can be delivered incrementally, variance analysis options, and requirements for the Performance Narrative Report content.

What is included in the IPMDAR deliverable?

There are three components as outlined below

IPMDAR Components

Notes:

  1. At a minimum, the IPMDAR requires data at the control account level with summary element of cost detail. The contract may specify work package level data.
  2. Inputs from a recent schedule risk assessment (SRA) should be included in the native schedule file submission when available. Depending on the schedule tool, the SRA data may need to be a separate file submission (Word or PDF). Results from the SRA along with other schedule analysis discussions (critical path, driving path, and schedule margin) are required to be included in the Detailed Analysis Report narrative. 
  3. The customer may request the results from a schedule data quality assessment and health metrics be included in the Detail Analysis Report narrative. 

What is required to produce the IPMDAR deliverables? 

For contractors new to EVM, one of their first objectives is to figure out what schedule and cost tools they need to be able to provide the required IPMDAR data and narrative analysis to their customer. H&A earned value consultants are sometimes asked to provide recommendations on commercial off the shelf (COTS) tools for this purpose. Much depends on what the contractor already has in place. 

Common schedule COTS tools such as MSP or Oracle Primavera P6 that have already been implemented will require an add-on to produce the SPD. Keep in mind that the IPMDAR does require SRA data and may require results from performing routine schedule data quality assessments. Some COTS add-ons to MSP or P6 are able to produce the typical schedule data quality metrics as well as produce the SPD. Other COTS scheduling tools such as Deltek Open Plan incorporate the SRA functionality, data quality metrics, and the ability to produce the SPD as part of the core product capabilities.

In most instances, contractors new to EVM do have an accounting/financial system in place to at least capture some level of contract or program/project budget and actual cost data. There may also be some capability to produce ETC data required for EAC financial analysis. The issue is organizing the complete set of time phased cost data (budget, earned value, actual cost, ETC) at the control account and work package level by summary element of cost that aligns with the schedule activity data. A contractor may be able to get by with Excel for a small project, however, it is time and cost prohibitive to create Excel macros to produce the CPD. Most COTS EVM cost tools are able to produce the CPD and have successfully completed the DoD EVM-CR data submission validation checks. This is a better alternative to building an in-house tool.

A data analysis tool such as Encore Analytics Empower is also a good option. Empower can import the time phased cost data from Excel or other COTS EVM tools and produce the CPD output. Empower can also import data from common COTS schedule tools. The benefit to using Empower is the ability to analyze the schedule and cost data in one place to verify alignment, produce interactive dashboards and a variety of analysis data views, and produce the IPMDAR Performance Narrative Executive Summary and Detailed Analysis Report.

Top Three Tips for Implementing Tools to Produce the IPMDAR Outputs

Here are a few tips from H&A’s earned value consultants on implementing tools to support the IPMDAR data submittals. Focus on getting the basics right.

  • Continuously verify the quality of the schedule and cost data. Routinely perform schedule data quality assessment and health checks to proactively resolve schedule construction, status, or data issues. Perform routine cost data validation checks such as earned value and no actual costs for a work package or the cumulative to date earned value exceeds the budget at completion (BAC). Correct all data anomalies before producing the deliverables.
  • Continuously verify the schedule and cost data are in alignment. Consistent schedule and cost data coding is critical to ensure integration and traceability. 
  • Anticipate the scope and level of data detail required. This can impact tool configuration, data structures, and data pulled from other business systems such accounting. For example, be prepared to provide the work package level of detail; actual costs will need to be available at this level. Another example is providing schedule risk assessment inputs; this is usually required at intervals specified in the CDRL.

H&A earned value consultants routinely help clients with constructing the schedule to support the IPMDAR data requirements, setting up the process to do routine data quality checks, integrating the schedule and cost data, and verifying the data before producing the performance reporting data submittals. Another common focus is producing clear and concise variance analysis narrative content. 

We can do the same for you. Call us today at (714) 685-1730 to get started.

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs Read Post »

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

By / /
Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

IEAC_Study_Project_Data_Template-Rev-06052023Download

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs

By / / , , ,
, , ,

A previous blog, Maintaining a Credible Estimate at Completion (EAC), discussed why producing a realistic EAC is essential to managing the remaining work on a contract. Internal management and the customer need visibility into the most likely total cost for the contract at completion to ensure it is within the negotiated contract cost and funding limits.

As noted in the earlier blog, one common technique to test the realism of the EAC is to compare the cumulative to date Cost Performance Index (CPI) to the To Complete Performance Index (TCPI).

Example of Using the Metrics for Evaluating Data

One example of documented guidance to industry for evaluating the realism of the EAC is the DOE Office of Project Management (PM) Compliance Assessment Governance (CAG) 2.0, and the related DOE EVMS Metric Specifications they use to assess the quality of schedule and cost data. This blog highlights the use of this guidance and how any contractor can incorporate similar best practices to verify EACs at a given WBS element, control account, or project level are realistic.

To refresh, the CPI is the efficiency at which work has been performed so far for a WBS element, control account, or at the total project level. The formula for the cumulative to date CPI is as follows.

Best practice tip: To ensure a valid CPI calculation, verify the BCWP and ACWP are recorded in the same month for the same work performed.

The TCPI provides the same information, however, it is forward looking. While the CPI is the work efficiency so far, the TCPI is the efficiency required to complete the remaining work to achieve the EAC. The formula for the TCPI is as follows.

TCPI Formula

Best practice tip: To ensure a valid TCPI, verify the BCWP and ACWP are recorded in the same month for the same work performed, and the BAC and EAC are for the same work scope. In other words, the scope of work assumptions are the same for the budget and remaining cost. This is why anticipated changes should not be included in the EAC.

The DOE uses the CPI in two of their assessment metrics and the TCPI in one, however, these are critical metrics partly because they are the only ones used to assess two different data evaluations: 1) commingling level of effort (LOE) and discrete work, and 2) EAC realism.

Commingling LOE and Discrete Work

The first use of CPI (no TCPI in this metric) falls under the Budgeting and Work Authorization subprocess. The primary purpose is to evaluate the effect of commingling LOE and discrete work scope has on control account metrics. The basic premise for this metric is that if the CPI for the LOE scope is significantly different than that for the discrete, the mixture of LOE in that control account is likely skewing overall performance reporting.

Here is the formulation DOE uses.

C.09.01:  Control Account CPI delta between Discrete and LOE >= ±0.1

X = Number of incomplete control accounts (WBS elements) in the EVMS cost tool, where

  1. The LOE portion of the budget is between 15% and 80% of the total budget, and
  2. The difference between the CPI for the discrete work and the LOE work is >= ±0.1.
Y = Number of incomplete control accounts (WBS elements) in the EVMS cost tool.
Threshold = 0%

Best practice tip: Run this metric quarterly on your control accounts that commingle LOE and discrete work packages. When there is a significant discrepancy between the performance of the LOE versus discrete work effort, consider isolating the LOE effort from the discrete effort at the earliest opportunity. An example could be the next rolling wave planning window or as part of an internal replanning action. Alternatively, it may be necessary to perform the calculations at the work package level to assess the performance of just the discrete effort when it is impractical to isolate by other means.

Process and procedure tip: Ensure the LOE work packages within a control account are kept to minimum (typically less than 15%), during the baseline development phase. This helps to prevent discrete work effort performance measurement distortion during the execution phase. A useful best practice H&A earned value consultants have helped contractors to implement during the budget baseline development process is to perform an analysis of the earned value methods used within a control account and the associated work package budgets. This helps to verify any LOE work packages are less than the 15% threshold for the control account. In some instances, it may be logical to segregate the LOE work effort into a separate control account. The objective is to identify and resolve the issue before the performance measurement baseline (PMB) is set.

EAC Realism

One DOE metric uses the TCPI and this involves a comparison to the CPI. This falls in the Analysis and Management Reporting subprocess. This DOE EVMS Metric Specification states: “This metric confirms that estimates of costs at completion are accurate and detailed.” As noted above, the metric compares the cost performance efficiency so far to the cost efficiency needed to achieve the EAC and is specific to the EAC a control account manager (CAM) would review for their scope of work. Depending on the level actual costs are collected, this analysis may need to be performed at the work package level instead of the control account level.  

Here is the formulation DOE uses assuming actual costs are collected at the work package level.

F.05.06:  Work Package CPI – EAC TCPI > ±0.1
X = Number of incomplete (>10% complete) work packages where CPI –TCPI > ±0.1.
Y = Number of incomplete (>10% complete) work packages in the EVMS cost tool.
Threshold = 5%

There is no requirement that the forecast of future costs has a linear relationship with past performance. While there may be legitimate reasons why future cost performance will fluctuate from the past, outside reviewers who receive EVM data will look for a trend or preponderance of data that would indicate the EACs are not realistic. When a significant number of active work packages are outside the ±0.1 CPI-TCPI threshold, it is an indication that the EACs are not being maintained or are driven by factors other than project performance.

Best practice tip: Run this metric every month for each active work package prior to month-end close. For those work packages outside the ±0.1 threshold, review the EAC to ensure it is an intentional forecast of costs given the current conditions.

Process and procedure tip: One of the training courses H&A earned value consultants often conduct is a Variance Analysis Reporting (VAR) workshop. This workshop is designed to help CAMs become more proficient with using the EVM metrics to assess the performance to date for their work effort, identify the root cause of significant variances, and document their findings as well as recommended corrective actions. This analysis includes verifying their estimate to complete (ETC) is a reasonable assessment of what is required to complete the remaining authorized work and their EACs are credible.

 

Additional References

Further discussion on using the CPI and TCPI to assess the EAC realism at the project level can be found in the DOE CAG, Analysis and Management reporting subprocess, Estimates at Completion. This section provides a good overview of comparing the cumulative to date CPI to the TCPI as well as comparing an EAC to calculated independent EACs (IEACs) for further analysis to assess the EAC credibility. 

Interested in learning more about using EVM metrics as a means to verify EACs at the detail or project level are realistic? H&A earned value consultants can help you incorporate best practices into your processes and procedures as well as conduct targeted training to improve your ETC and EAC process. Call us today at (714) 685-1730.

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs Read Post »

How Integrated Baseline Reviews (IBRs) Contribute to Project Success

By / / , , , ,
, , , ,

What is an Integrated Baseline Review or IBR?

An IBR is a formal review of a contractor’s performance measurement baseline (PMB) a customer conducts shortly after contract award or other project events to gain confidence in the contractor’s ability to deliver and meet contract objectives. Conducting an IBR helps to assure there is mutual agreement on the scope of work, schedule, resource requirements, and budget to meet the customer’s needs. It also assures there is a mutual understanding of the project’s risks and opportunities as well as how they will be managed.

Conducting an IBR is often a contractual requirement along with the requirement to implement an earned value management system (EVMS). Contractual documents specify the time frame for when the IBR must occur after contract award. This is typically within 90 to 180 calendar days. A customer may also conduct an IBR at critical milestones, funding gates, when transitioning to another project phase, or when significant changes are incorporated into a PMB.

What an IBR is Not

An IBR is not an EVMS compliance review. The intent of an IBR is not to devolve into a review of the contractor’s EVMS and whether it complies with the EIA-748 Standard for EVMS guidelines. That said, the contractor must be able to demonstrate they have a disciplined project control system in place. The contractor should be able to demonstrate to the customer that the project’s scope of work is properly planned, scheduled, resourced, budgeted, authorized, and managed using their project control system.

What are the benefits of conducting an IBR?

Conducting an IBR contributes to successful project execution because it helps to ensure a realistic PMB has been established.

IBRs provide the opportunity for the contractor and customer to verify:

  • There is shared understanding of the scope of work, technical requirements, and accomplishment criteria. As the work breakdown structure (WBS) is decomposed into manageable product-orientated work elements, it provides a common frame of reference for communication between the contractor and customer. The WBS dictionary should capture the technical requirements that must be met as well as expected deliverables and outcomes. The contractor must have a clear understanding of customer’s needs, assumptions, and expectations to be able to create a realistic schedule and budget plan. The IBR provides the opportunity for the contractor to verify the scope of work details with the customer before the project execution phase begins. In instances where the technical requirements evolve over time as work progresses, rolling wave planning is often used to detail plan the current work effort with more macro planning for future work effort to ensure the entire scope of work is included.
  • An executable PMB has been established for the entire contractual scope of work. The PMB should accurately reflect how the contractor plans to accomplish the work within the contractual period of performance and negotiated contract cost. The customer’s funding profile may also determine the timing of activities and when resources are required. The schedule and budget should be in alignment. The budget time phasing should reflect the schedule activities and resource requirements. It is also useful to verify appropriate earned value methods and techniques have been selected for the work packages to assure objective and meaningful project performance can be measured and reported as work progresses.
  • The required resources have been identified and assigned to the project. This contributes to producing an executable schedule and budget plan. The staffing plan should accurately reflect the sequence of work and skill mix as well as resource availability and demand to accomplish the project’s objectives. Flat loading labor hours may not accurately reflect common challenges of ramping up resources after contract award or the availability of critical resources for specific tasks. Other resource factors include the timing or availability of critical or high value materials as well as subcontractors responsible for performing work or providing services.
  • Project technical, schedule, and cost risks/opportunities have been identified and assessed. This also contributes to producing an executable schedule and budget plan. Where possible, risk mitigation actions have been incorporated into the PMB to reduce known risks to an acceptable level. For example, the timing or duration of activities as well as resource requirements may need to be adjusted. Schedule margin activities may be incorporated into the integrated master schedule (IMS). It also provides fact-based information to determine the amount of management reserve set aside to handle realized risks. This is often the most valuable component of the IBR. It is essential all parties have an understanding of the identified risks or opportunities, potential impact if they are realized, and risk mitigation or opportunity capture plans.

Why it is important to verify these details during an IBR?

A realistic schedule and budget plan helps to prevent cost growth surprises because of technical, schedule, or budget challenges. The better the up-front planning, the less the likelihood of cost growth during project execution. It also increases credibility with the customer. The contractor can demonstrate their ability to deliver to the customer needs and manage the work effectively.

Benefits of Preparing for an IBR

Establishing a project’s PMB is a significant and often formal event as it signals the transition from the planning to execution phase. It represents the culmination of the integrated planning, scheduling, budgeting, work authorization, and risk/opportunity management processes.

A common best practice is to conduct an internal baseline review regardless of whether a formal IBR with the customer is required prior to setting the PMB. Implementing a standard process to conduct an internal review of the complete set of project data and artifacts with the project personnel assures an executable schedule and budget plan has been established to accomplish the contractual scope of work within the contractual period of performance and negotiated contract cost in alignment with the contract’s funding profile.

These internal reviews help to ensure there is a common understanding of the scope of work, major project events, planned sequence of work, schedule of deliverables, resource requirements, time phased budget, funding profile, and project risks/opportunities. It also provides an opportunity to verify the quality of the integrated schedule and cost data as well as top down and bottom up traceability. 

Need help preparing for an IBR?

A common earned value consulting service H&A provides is conducting a mock IBR with project personnel to prepare for the formal customer IBR. The objective is to conduct a thorough assessment of the project’s PMB to verify it reflects the entire contractual scope of work and technical requirements as well as identified technical, schedule, cost, or resource risks that may impact the ability to execute the work as planned. This provides an opportunity to correct any issues with the PMB prior to the IBR event.

Another standard earned value consulting service we offer is conducting IBR training for project team members. H&A earned value consultants can help you to establish a standard internal process to verify an executable PMB is in place for a given project. Once again, the objective is to prevent cost growth surprises and management is aware of the project’s risks and opportunities that may impact profit margins. 

Call us today at (714) 685-1730 to get started. 

How Integrated Baseline Reviews (IBRs) Contribute to Project Success Read Post »

EVMS Guidelines 2 – Five Sections of Organizational Guidelines

By / / , , ,
, , ,

Video Contents

The EVMS Guidelines are Grouped in five major categories which address these five sections:

0:12 – Organization
0:32 – Planning, Scheduling, and Budgeting
0:46 – Accounting Considerations
0:56 – Analysis and Management Reports
1:15 – Revisions and Data Maintenance

More EVMS Training

If you liked this video you can purchase the entire course below. This video is an excerpt from the Department of Defense (DOD) version of this eLearning module. We also offer the same course customized for the Department of Energy’s (DOE) specific Earned Value Management (EVM) implementation/requirements, as well as a version of the course customized for NASA’s EVM implementation/requirements.  

— Purchase This Course —
EVMS DOD Virtual Learning Lab

— Purchase the DOE Version of this Course —
EVMS DOE Virtual Learning Lab

— Purchase the NASA Version of this Course —
EVMS NASA Virtual Learning Lab

EVMS Document Matrix

EVMS Document Matrix

Not sure what the different requirements are between the DOE and NASA? Can’t remember if Cost and Software Data Reporting (CSDR) is required for an NSA contract? Check out our easy to read Earned Value Management Systems Document Matrix

All Online Courses

All Online Courses Available from Humphreys & Associates

Earned Value Training

EVMS Guidelines 2 – Five Sections of Organizational Guidelines Read Post »

Scroll to Top