Earned Value Management System (EVMS)

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

By / / ,
,
Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

IEAC_Study_Project_Data_Template-Rev-06052023Download

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Tips for Producing an Earned Value Management System Description

By / / , ,
, ,
Tips for Producing an EVM System Description Blog Post Banner Image.

There are a variety of ways a contractor can produce the documentation that describes their Earned Value Management System (EVMS), how it is implemented, used, and maintained. This is typically referred to as the EVM System Description (SD), though other names may be used. For example, it could be titled the Integrated Project Management System Description or Project Controls System Description to align with corporate or contractual naming conventions. EVM practices may be just one part of their overall project management system.

Regardless of the title, a contractor with an EVMS contractual requirement must be able to provide the necessary documentation to their customer to demonstrate their system complies with the EIA-748 Standard for EVMS guidelines. This includes any additional implementation guidance associated with the government agency responsible for conducting EVMS certification or validation reviews.

What is the purpose of an EVM System Description?

There are two purposes for an EVM SD; one is internal, the other is for the external customer.

The primary user of the EVM SD is project personnel such as project managers, project controls, schedulers, and control account managers (CAMs) responsible for implementing the EVMS on their project. It is essential they have a clear understanding of the required corporate EVM process and procedures as well as how to meet the intent of the EIA-748 guidelines using the applicable schedule, cost, analysis, risk, and other project control toolsets.

The government customer uses the EVM SD to gain an understanding of the contractor’s EVMS. The EVM SD must adequately document how the contractor’s system meets the intent of the EIA-748 guidelines. It should include a map of the EVM SD content to the EIA-748 guidelines or similar government customer checklist. The purpose of a government agency’s compliance review is to conduct a comprehensive assessment of a contractor’s EVMS. This includes the EVM SD as well as related processes and procedures. They also assess how the contractor has implemented the EVMS to verify the EVMS is providing timely, reliable, and auditable information.

Once the Cognizant Federal Agency (CFA) has determined the contractor’s EVMS complies with the EIA-748 guidelines, they conduct routine surveillance to ensure the contractor’s system as documented in the EVM SD and implemented on projects continues to comply with the EIA-748 guidelines. The contractor’s EVM SD is also a typical artifact required for an Integrated Baseline Review (IBR).

What needs to be included in the EVM SD?

An EVM SD should explain the methodology the contractor uses to comply with the EIA-748 guidelines as well as any government agency specific requirements. For example, DCMA uses the DoD EVMS Interpretation Guide (EVMSIG) and their Earned Value Management System Center Business Practice (BP) documents such as the BP 2 EVM System Description Review attachment EVMS Cross Reference Checklist (CRC). The DOE Office of Project Management (PM) uses their EVMS Compliance Review Standard Operating Procedure (ECRSOP) and related appendices including their Compliance Assessment Governance (CAG) document and EVMS Compliance Reference Crosswalk (CRC).

The EVM SD should provide an executive overview, reference the corporate EVMS policy, and address all of the subsystems that make up the EVMS. H&A earned value consultants have reviewed countless system descriptions over the years; contractors take a variety of approaches to discuss their subsystems.

Some contractors align with the EIA-748 five process areas (Organizing; Planning, Scheduling, and Budgeting; Accounting; Analysis and Management Reports; and Revisions and Data Maintenance). Others include additional sections covering indirect costs, material management, and subcontractor management. The planning, scheduling, and budgeting process area is often broken down into two sections – one for planning and scheduling and the other to cover work authorization and budgeting (a total of nine process groups). For more discussion, see the blog on the Benefits of Using Nine Process Groups.

A DOE contractor may align with the DOE CAG 10 subprocesses; this is similar to the nine progress group approach plus a section on risk and opportunity management. Other contractors use the project life cycle approach (initiation, planning, execution, monitor and control, and close out) to organize their content.

We recommend including a section on self-governance or self-surveillance (see Benefits of an EVMS Self-Governance Process). As noted in this blog, contractors with an approved EVMS are expected to establish and execute an annual EVMS self-governance plan.

The government customer expects contractors to map their EVM SD content to the applicable government agency EVMS compliance checklist. This is typically an appendix to the EVM SD. In most instances, this is the DCMA BP 2 EVMS Cross Reference Checklist or the DOE ECRSOP EVMS Compliance Reference Crosswalk. These checklists include attributes and/or questions that provide a more comprehensive guide to assess how a contractor meets the intent of the guidelines. The government customer also uses these checklists to identify and record the areas in the system documentation that meet the intent or note deficiencies the contractor needs to address.

The system description should include a concise description of what’s required, diagrams, illustrations, process flowcharts, as well as sample forms and reports with example data. In preparation for a compliance review, H&A earned value consultants often assist clients in creating an EVMS storyboard that visually depicts the entire system using the workflow diagrams along the inputs and outputs, and who is doing what using the applicable tool that demonstrates the system in operation.

Single or multiple documents?

A common question H&A earned value consultants are asked is whether the EVM SD should be a single document that includes the complete set of flowcharts, forms, and reports or a summary document with supporting procedures. Typically, the supporting procedures include the process flowcharts and sometimes the applicable artifacts (forms and reports). There are pros and cons to each approach.

Single Document Approach
Pros Cons
  • Provides an integrated view of the entire EVMS process.
  • Provides a complete discussion of a topic without having to reference multiple documents. Requirement discussion and process flowcharts with responsibility swim lanes along with example inputs/outputs provides a complete picture.
  • Configuration control is easier to manage, only need to make changes in one place.
  • Easier to search for a topic in a single document.
  • Easier to map the contents of a single document to the DCMA or DOE CRC.
  • Easier to create and maintain cross reference links between sections within a single document (See or See Also references).
  • For a company new to EVM, can appear to be an overwhelming task to produce a single, comprehensive document.
  • Can be a lengthy document.
  • More difficult to partition the content to different process owners to complete their section and to combine the content.
Summary Document with Supporting Procedures Approach
Pros Cons
  • Shorter summary document provides a general overview as a quick introduction to the EVMS. Can appear less daunting. Can be used as a strategy to introduce EVM concepts and incorporate into standard business practices.
  • Modular approach to creating and maintaining content. Can be easier to create the initial content.
  • Useful for targeted training using the individual procedures for a process area. For example, training focused on the steps required for developing the WBS and WBS dictionary with expected outputs.
  • Someone must review multiple documents to gain a complete understanding of the EVMS.
  • Fragmented and potentially siloed view of the system.
  • Potentially larger volume of content (over a single document), content is often repeated for context.
  • Configuration control. More difficult to maintain content to ensure consistency in multiple documents as well as maintain cross references between documents.
  • More difficult to map content in multiple documents to the DCMA or DOE CRC.

Regardless of which approach you take, keep in mind that the government customer will be approving the complete set of EVM SD documents (one or multiple documents). Any document referenced in the EVM SD is also subject to review. When you make changes to your EVM SD, the government customer will need to review and approve those changes to ensure continuing compliance with the EIA-748 guidelines. This is a formal process; changes must be processed through the contracting officer. 

Need help with your EVM System Description?

Whether you need to update your existing EVM SD or create one, H&A earned value consultants can help you to organize the content and artifacts in alignment with your business requirements. They also work with you to ensure your EVMS satisfies the EIA-748 guideline requirements as well as government agency specific requirements. H&A often assists with designing an EVMS and ensuring the project control software tools are configured to support the EVMS as well as verifying the quality of the data. For clients new to EVM, H&A offers a template that provides the basis to develop an EVM SD. This template is intended to assist clients with designing and implementing an EVMS that meets EIA-748 guideline requirements as well as the more rigorous requirements of specific government customers.

Call us today at (714) 685-1730 to get started.

Tips for Producing an Earned Value Management System Description Read Post »

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs

By / / , , ,
, , ,

A previous blog, Maintaining a Credible Estimate at Completion (EAC), discussed why producing a realistic EAC is essential to managing the remaining work on a contract. Internal management and the customer need visibility into the most likely total cost for the contract at completion to ensure it is within the negotiated contract cost and funding limits.

As noted in the earlier blog, one common technique to test the realism of the EAC is to compare the cumulative to date Cost Performance Index (CPI) to the To Complete Performance Index (TCPI).

Example of Using the Metrics for Evaluating Data

One example of documented guidance to industry for evaluating the realism of the EAC is the DOE Office of Project Management (PM) Compliance Assessment Governance (CAG) 2.0, and the related DOE EVMS Metric Specifications they use to assess the quality of schedule and cost data. This blog highlights the use of this guidance and how any contractor can incorporate similar best practices to verify EACs at a given WBS element, control account, or project level are realistic.

To refresh, the CPI is the efficiency at which work has been performed so far for a WBS element, control account, or at the total project level. The formula for the cumulative to date CPI is as follows.

Best practice tip: To ensure a valid CPI calculation, verify the BCWP and ACWP are recorded in the same month for the same work performed.

The TCPI provides the same information, however, it is forward looking. While the CPI is the work efficiency so far, the TCPI is the efficiency required to complete the remaining work to achieve the EAC. The formula for the TCPI is as follows.

TCPI Formula

Best practice tip: To ensure a valid TCPI, verify the BCWP and ACWP are recorded in the same month for the same work performed, and the BAC and EAC are for the same work scope. In other words, the scope of work assumptions are the same for the budget and remaining cost. This is why anticipated changes should not be included in the EAC.

The DOE uses the CPI in two of their assessment metrics and the TCPI in one, however, these are critical metrics partly because they are the only ones used to assess two different data evaluations: 1) commingling level of effort (LOE) and discrete work, and 2) EAC realism.

Commingling LOE and Discrete Work

The first use of CPI (no TCPI in this metric) falls under the Budgeting and Work Authorization subprocess. The primary purpose is to evaluate the effect of commingling LOE and discrete work scope has on control account metrics. The basic premise for this metric is that if the CPI for the LOE scope is significantly different than that for the discrete, the mixture of LOE in that control account is likely skewing overall performance reporting.

Here is the formulation DOE uses.

C.09.01:  Control Account CPI delta between Discrete and LOE >= ±0.1

X = Number of incomplete control accounts (WBS elements) in the EVMS cost tool, where

  1. The LOE portion of the budget is between 15% and 80% of the total budget, and
  2. The difference between the CPI for the discrete work and the LOE work is >= ±0.1.
Y = Number of incomplete control accounts (WBS elements) in the EVMS cost tool.
Threshold = 0%

Best practice tip: Run this metric quarterly on your control accounts that commingle LOE and discrete work packages. When there is a significant discrepancy between the performance of the LOE versus discrete work effort, consider isolating the LOE effort from the discrete effort at the earliest opportunity. An example could be the next rolling wave planning window or as part of an internal replanning action. Alternatively, it may be necessary to perform the calculations at the work package level to assess the performance of just the discrete effort when it is impractical to isolate by other means.

Process and procedure tip: Ensure the LOE work packages within a control account are kept to minimum (typically less than 15%), during the baseline development phase. This helps to prevent discrete work effort performance measurement distortion during the execution phase. A useful best practice H&A earned value consultants have helped contractors to implement during the budget baseline development process is to perform an analysis of the earned value methods used within a control account and the associated work package budgets. This helps to verify any LOE work packages are less than the 15% threshold for the control account. In some instances, it may be logical to segregate the LOE work effort into a separate control account. The objective is to identify and resolve the issue before the performance measurement baseline (PMB) is set.

EAC Realism

One DOE metric uses the TCPI and this involves a comparison to the CPI. This falls in the Analysis and Management Reporting subprocess. This DOE EVMS Metric Specification states: “This metric confirms that estimates of costs at completion are accurate and detailed.” As noted above, the metric compares the cost performance efficiency so far to the cost efficiency needed to achieve the EAC and is specific to the EAC a control account manager (CAM) would review for their scope of work. Depending on the level actual costs are collected, this analysis may need to be performed at the work package level instead of the control account level.  

Here is the formulation DOE uses assuming actual costs are collected at the work package level.

F.05.06:  Work Package CPI – EAC TCPI > ±0.1
X = Number of incomplete (>10% complete) work packages where CPI –TCPI > ±0.1.
Y = Number of incomplete (>10% complete) work packages in the EVMS cost tool.
Threshold = 5%

There is no requirement that the forecast of future costs has a linear relationship with past performance. While there may be legitimate reasons why future cost performance will fluctuate from the past, outside reviewers who receive EVM data will look for a trend or preponderance of data that would indicate the EACs are not realistic. When a significant number of active work packages are outside the ±0.1 CPI-TCPI threshold, it is an indication that the EACs are not being maintained or are driven by factors other than project performance.

Best practice tip: Run this metric every month for each active work package prior to month-end close. For those work packages outside the ±0.1 threshold, review the EAC to ensure it is an intentional forecast of costs given the current conditions.

Process and procedure tip: One of the training courses H&A earned value consultants often conduct is a Variance Analysis Reporting (VAR) workshop. This workshop is designed to help CAMs become more proficient with using the EVM metrics to assess the performance to date for their work effort, identify the root cause of significant variances, and document their findings as well as recommended corrective actions. This analysis includes verifying their estimate to complete (ETC) is a reasonable assessment of what is required to complete the remaining authorized work and their EACs are credible.

 

Additional References

Further discussion on using the CPI and TCPI to assess the EAC realism at the project level can be found in the DOE CAG, Analysis and Management reporting subprocess, Estimates at Completion. This section provides a good overview of comparing the cumulative to date CPI to the TCPI as well as comparing an EAC to calculated independent EACs (IEACs) for further analysis to assess the EAC credibility. 

Interested in learning more about using EVM metrics as a means to verify EACs at the detail or project level are realistic? H&A earned value consultants can help you incorporate best practices into your processes and procedures as well as conduct targeted training to improve your ETC and EAC process. Call us today at (714) 685-1730.

Using Earned Value Management (EVM) Performance Metrics for Evaluating EACs Read Post »

Charts and EVMS Health

By / / ,
,

Project management is a complex and multi-faceted field that requires a great deal of attention to detail. One of the most important aspects of project management is being able to accurately assess the status and health of a project at any given time. This is where visual representations come in.

Visual Aids

One of the most effective ways to evaluate the status of a project is through the use of charts and graphs. These visual aids allow top management to quickly and easily assess whether a project is getting better or worse, and whether the estimated completion costs (EACs) are achievable.

Raising Questions

In the case of the chart shown in the video, the cost line is trending downwards, which is a cause for concern. This raises the question of whether this negative trend can be reversed to meet even the most pessimistic EAC set by the customer’s program manager.

Management and Customers

While the chart shown in the video is from a program that was ultimately canceled, it serves as an important reminder of the crucial role that visual representations play in project management. These types of charts and graphs are commonly used in briefs presented to top management and customers, as they provide a clear and concise snapshot of the project’s status.

More EVMS Training

Take your EVMS training to the next level with our online course that is based on Humphreys & Associates’ highly regarded three-day EVMS workshop. We also offer a version of this same EVM training course customized for Department Of Defense (DOD) specific EVM requirements, as well as one designed specifically toward NASA’s EVM requirements!

— Purchase the DOD Course —
EVMS DOD Virtual Learning Lab

— Purchase the DOE Version of this Course —
EVMS DOE Virtual Learning Lab

— Purchase the NASA Version of this Course —
EVMS NASA Virtual Learning Lab

EVMS Document Matrix

EVMS Document Matrix

Not sure what the different requirements are between the DOE and NASA? Can’t remember if Cost and Software Data Reporting (CSDR) is required for an NSA contract? Check out our easy to read Earned Value Management Systems Document Matrix

All Online Courses

EVM (Earned Value Management) is a comprehensive approach to project management that covers all aspects of a project, from start to finish. Humphreys and Associates offers EVM courses that will provide you with the necessary skills and knowledge to manage EVM projects of all sizes effectively.

Our EVM courses are suitable for both beginners and professionals who want to improve their skills. We offer a range of course options, including online self-paced and live virtual instructor-led courses, to accommodate different schedules and learning styles. Our instructors are seasoned EVM professionals who will guide you through the process, from setting up your EVM system to closing out your project.

At Humphreys and Associates, you can trust in the quality of our EVM training. Our online EVM courses are budget-friendly and flexible, allowing you to obtain the education you need without any stress. Plus, our courses are designed to be completed in a short time, so you can quickly gain the knowledge you need.

Enroll in our EVM courses today and start enhancing your project management skills.

All Online Courses Available from Humphreys & Associates

Upcoming Public EVMS Workshops

EVM training sessions are offered throughout the year by H&A. EVMS, advanced Earned Value Management techniques, and project scheduling are all covered during these frequent public EVM workshops. These earned value education courses are ideal for anybody wanting to increase their understanding of or application of EVMS or scheduling. Our training programs are suited for individuals with various degrees.

Public EVMS Workshops are offered either in person or online. View our Upcoming EVM Workshop Schedule to find one that best suits your schedule and location.

Upcoming Earned Value Training Public Workshops

Charts and EVMS Health Read Post »

Top Management Use of Earned Value Data

By /

Top management relies on performance measurement data to make informed decisions about whether a project is worth undertaking and how much money should be allocated to it. This is done by examining trends in performance data over time and benchmarking the organization against others. Additionally, top management uses performance measurement data to assess the effectiveness of their own interventions to improve organizational performance.

Project complexity is increasing and costs are soaring. Despite this, top management remains ever vigilant in their pursuit of ensuring that taxpayer money is being well spent on government projects. In this lesson, we will explore how all performance measurement data are used by top management to make informed decisions about government projects. Stay tuned!

More EVMS Training

If you liked this video you can purchase the entire course below. This video is an excerpt from the Department of Defense (DOD) version of this eLearning module. We also offer the same course customized for the Department of Energy’s (DOE) specific Earned Value Management (EVM) implementation/requirements, as well as a version of the course customized for NASA’s EVM implementation/requirements.  

— Purchase This Course —
EVMS DOD Virtual Learning Lab

— Purchase the DOE Version of this Course —
EVMS DOE Virtual Learning Lab

— Purchase the NASA Version of this Course —
EVMS NASA Virtual Learning Lab

EVMS Document Matrix

EVMS Document Matrix

Not sure what the different requirements are between the DOE and NASA? Can’t remember if Cost and Software Data Reporting (CSDR) is required for an NSA contract? Check out our easy to read Earned Value Management Systems Document Matrix

All Online Courses

EVMS courses are available online, making it easy for you to get the education and training you need without having to leave your home or office. Plus, our instructors are experienced professionals who will help you learn everything you need to know about EVMS.

We have a wide range of courses available, from introductory level to more advanced subjects. You can find the perfect course for your needs and start learning today.

Our online EVMS courses are affordable and convenient, so you can get the education you need without any hassle. Plus, our courses are designed to help you learn everything you need to know about EVMS in a short amount of time.

Get started today and find the perfect online EVMS course for you.

All Online Courses Available from Humphreys & Associates

Upcoming Public EVMS Workshops

EVM training sessions are offered throughout the year by H&A. EVMS, advanced earned value management techniques, and project scheduling are all covered during these frequent public EVM workshops. These earned value education courses are ideal for anybody wanting to increase their understanding of or application of EVMS or scheduling. Our training programs are suited for individuals with various degrees.

Public EVMS Workshops are offered either in person or online. View our Upcoming EVM Workshop Schedule to find one that best suits your schedule and location.

Upcoming Earned Value Training Public Workshops

Top Management Use of Earned Value Data Read Post »

EVMS Guidelines 2 – Five Sections of Organizational Guidelines

By / / , , ,
, , ,

Video Contents

The EVMS Guidelines are Grouped in five major categories which address these five sections:

0:12 – Organization
0:32 – Planning, Scheduling, and Budgeting
0:46 – Accounting Considerations
0:56 – Analysis and Management Reports
1:15 – Revisions and Data Maintenance

More EVMS Training

If you liked this video you can purchase the entire course below. This video is an excerpt from the Department of Defense (DOD) version of this eLearning module. We also offer the same course customized for the Department of Energy’s (DOE) specific Earned Value Management (EVM) implementation/requirements, as well as a version of the course customized for NASA’s EVM implementation/requirements.  

— Purchase This Course —
EVMS DOD Virtual Learning Lab

— Purchase the DOE Version of this Course —
EVMS DOE Virtual Learning Lab

— Purchase the NASA Version of this Course —
EVMS NASA Virtual Learning Lab

EVMS Document Matrix

EVMS Document Matrix

Not sure what the different requirements are between the DOE and NASA? Can’t remember if Cost and Software Data Reporting (CSDR) is required for an NSA contract? Check out our easy to read Earned Value Management Systems Document Matrix

All Online Courses

All Online Courses Available from Humphreys & Associates

Earned Value Training

EVMS Guidelines 2 – Five Sections of Organizational Guidelines Read Post »

Scroll to Top